Skip to main content

Fast Data

During the 1980s and 1990s, online transaction processing (OLTP) was critical for banks, airlines, and telcos for core business functions. This was a big step-up from batch systems of the early days. We learnt the importance of sub-second response time and continuous availability with the goal of five-nines (99.999% uptime). The yearly tolerance of system outage was like 5 minutes. During my days at IBM, we had to face the fire from a bank in Japan that had an hour long outage resulting in a long queue in front of the ATM machine (unlike here, the Japanese stand very patiently until the system came back after what felt like an eternity). They were using IBM’s IMS Fast Path software and the blame was first put on that software, which subsequently turned out to be some other issue.

Advance the clock to today. Everything is real-time and one can not talk about real-time without discussing the need for “fast data” – data that has to travel very fast for real time decision making. Here are some reasons for fast data:

  • These days, it is important for businesses to be able to quickly sense and respond to events that are affecting their markets, customers, employees, facilities, or internal operations. Fast data enables decision makers and administrators to monitor, track, and address events as they occur.
  • Leverage the Internet of Things – for example, an engine manufacturer will embed sensors within its products, which then will provide continuous feeds back to the manufacturer to help spot issues and better understand usage patterns.
  • An important advantage that fast data offers is enhanced operational efficiency, since events that could negatively affect processes—such as inventory shortages or production bottlenecks—can not only be detected and reported, but remedial action can be immediately prescribed or even launched. Realtime analytical data can be measured against the patterns determined to predict problems, and systems can respond with appropriate alerts or automated fixes.
  • Assure greater business continuity – Fast data plays a role in bringing systems—and all data still in the pipeline—back up and running quickly, before the business suffers from a catastrophic event.
  • Fast data is critical for supporting Artificial Intelligence and machine learning. As a matter of fact, data is the fuel for machine learning (recommendation engines, fraud detection systems, bidding systems, automatic decision making systems, chatbots, and many more).

Now let us look at the constellation of technologies enabling fast data management and analytics. Fast data is the data that moves almost instantaneously from source to processing to analysis to action, courtesy of framework and pipelines such as Apache Spark, Apache Storm, Apache Kafka, Apache Kudu, Apache Cassandra, and in-memory data grids. Here is a brief outline on each of these.

Apache Spark – open source toolset now supported by most major database vendors. It offers streaming and SQL libraries to deliver real-time data processing. Spark Streaming offers data as it is created, enabling analysis for critical areas like real-time analytics and fraud detection. It’s structured streaming API opens up this capability to enterprises of all sizes.

Apache Storm is an open source distributed real-time computation system designed to enable processing of data streams.

Apache Cassandra is an open source low-latency data replication engine.

Apache Kafka is an open source toolset designed for real-time data streaming – employed for data pipelines and streaming apps. Kafka Connect API helps connect it to other environments. It originated at Linked-In.

Apache Kudu is an open source storage engine to support real-time analytics on commodity hardware.

In addition to powerful open source tools and frameworks, there are in-memory data grids that provides a hardware-enabled fast data enabler to deliver blazing speeds to meet today’s needs such as the IoT management and deployment of AI and machine learning and responding to events in real-time.

Yes, we have come a long way from those OLTP days! Fast data management and analytics is becoming a key area for businesses to survive and grow.

from: Jnan Dash’s Weblog
via Jnan Dash

Source: Fast Data Via Business Advice.

Comments

Post a Comment

Popular posts from this blog

How to Add Web Push Notification to Your WordPress Site

Do you want to add push notifications to your WordPress site? Push notifications allow you to send notifications to users even when they are not visiting your website. In this article, we will show you how to easily add web push notifications to your WordPress site. We will also talk about the best WordPress push notification plugins and how to send desktop & mobile push notifications from your WordPress site. What is Push Notification? Push notifications are clickable messages displayed on top of user’s desktop or notification area on their mobile device. They can be shown even when the user’s browser is not open. Aside from desktop, web push notifications also work on mobile devices. This allows you to reach your users across devices with latest updates and offers. Web push notifications have proven to be a very effective way to convert website visitors into customers and loyal followers. Why Add Web Push Notifications to Your WordPress Site? We have already discussed tha...
Post a Comment
Read more

7 Ways to Grow Your SaaS Startup Faster

Every startup looks for ways to catapult a business to success. Here are some tips for accelerating growth for SaaS (software as a service) companies. 1. Start charging early SaaS startups are often hesitant about charging customers. They think that their product is not yet ready, that it’s necessary to get traction and focus on expanding their customer base rather than on growing revenue. Deep inside, however, they often don’t just have enough faith in their product and are not sure if someone will buy it at all. They prefer keeping hundreds or thousands of free users to attempting to win a few serious customers. Big numbers are comforting but the product remains an unverified idea. Building a product should imply increasing revenue. So don’t hesitate too much—put it to the test by charging and see if it works in the real conditions. This is one way to minimize your risk too. Instead of waiting to launch until you’ve invested huge amounts of time and money, launch early. If...
Post a Comment
Read more

5 Things You Should Know about Purchasing a Company Vehicle

  Many businesses require a vehicle to do business. It transports both goods and people from point A to point B and allows you to keep business dealings in-house. However, choosing a good company vehicle can be somewhat complex. You shouldn’t take the decision lightly. Here are a few things you should know before you buy a vehicle for your business: Accidents can happen, placing liability on you. You might think that an accident won’t happen if you hire drivers with a good track record and write a rule-book with strict policies and procedures. These are good tactics for mitigating some of the risks associated with commercial car accidents, but they won’t protect you completely. More than two million car accidents happen each year. Many of them are commercial truck accidents, costing millions in damages. Since commercial drivers are on the road more than personal drivers, they’re significantly more likely to cause an accident. Understanding the liability that’s placed on yo...
Post a Comment
Read more