AWS Kinesis and AWS IoT services

Amazon Kinesis Service: is a platform for streaming data on AWS. It helps to load and analyze streaming data and build custom applications like:

• 1.     Recommendation engine.
• 2.     Real-time metering and billing system.
• 3.     Loading data from transactional databases to data warehouses can use Kinesis for temporary durable storage of data in transit.

Data generated by web applications, mobile devices, IoT sensors, wearable devices etc. can be in the tune of TBs/hr. This data needs to be collected in real-time and processed and stored continuously. Amazon Kinesis service enables one to do this.

The streaming data can be captured and submitted to Amazon Kinesis Firehose, which loads streaming data continuously into S3, Redshift and Amazon Elasticsearch domains. The streaming data is then analyzed from these stores using Business Intelligence tools.

Amazon Analytics: enables running standard SQL queries against data streams. Kinesis analytics can then be used to feed into other analytics tools.

AWS IoT: enables device to device and device to cloud applications communication. It can support billions of devices exchanging trillions of messages and can process and route those messages to AWS endpoints and to other devices reliably and securely. AWS IoT services can be used to build IoT applications that gather, process, analyze and act on data generated by connected devices, without having to manage any infrastructure.

It supports the following communication protocols:

1.     HTTP

2.     Websockets

3.     MQTT

The connection between devices and application are authenticated at AWS IoT service and only authenticated users are allowed to control the state of the device or the authenticated device connections to AWS IoT endpoints are allowed.

At AWS IoT service, one can filter, transform and act upon device data on the fly (say invoke a lambda function). The last state of device is persisted which can be read even when the device goes offline. The state change for the device can be requested while it is offline to be implemented on the device when it is online again by AWS IoT backend.

 How it works:

https://aws.amazon.com/iot/how-it-works/ 

Comparison of IoT  supported across cloud providers:

https://www.linkedin.com/pulse/azure-iot-hub-vs-aws-foundation-toby-mcclean 

Features	AWS IoT	IBM IoT Foundation	Microsoft IoT
Protocols	HTTP, MQTT, Websockets, custom protocols	HTTP, MQTT	HTTP, AMQP, MQTT and custom protocols
Certified Hardware Platforms	Broadcom, Marvell, Renesas, Texas Instruments, Microchip, Intel, Mediatek, QualComm, Seeed, BeagleBoard	ARM mbed, Texas Instruments, Raspberry Pi, Arduino Uno, Intel	Intel, Raspberry Pi 2, Freescale, Texas Instruments, MinnowBoard, BeagleBoard, Seeed, resin.io
Price	Paying million messages traffic (published from devices + delivered to devices)	Paying related to number of devices, data traffic and storage.	Paying for IoT Hub unit related to number of devices and messages per day
Communication Patterns	Telemetry, Command	Telemetry, Command	Telemetry, Command
SDK/Language	C, Node.js	C, C#, Python, Java, Node.js	.Net, Java, C, Node.js
Security	TLS (mutual auth)	TLS	TLS (sever only auth)
Authentication	X.509 certificate client authentication, IAM service, Cognito service	Per-device with token	Per-device with SAS token

What's Fog Computing?

Fog computing is an architecture that uses one or more of edge devices to carry out:

1.     a substantial amount of storage (instead of storing data in cloud),

2.     communication (instead of routing over internet),

3.     control, configuration and management (instead of being controlled by network gateways).

It is also referred to as “Fog networking” or “Fogging”.

The effects of fog computing on cloud computing and big data systems in common is to resolve the limitation in accurate content distribution. It is especially of significance for the numerous connected sensors in internet of things (IoT). For instance, semi-autonomous cars assist drivers in avoiding distractions and veering off the road by providing real-time analytics and decisions on driving patterns. Fog computing can also reduce the transfer of gigantic volumes of audio and video recordings generated by police dashboard and video cameras. Cameras equipped with edge computing capabilities could analyze video feeds in real time and only send relevant data to the cloud as needed.

Fog computing consists of a control and a data plane. On the data plane, fog computing enables computing services to reside at the edge of the network as opposed to servers in the datacenter.

Compared to cloud computing, fog computing emphasizes:

1.     proximity to end users

2.     dense geographical distribution

3.     local resource pooling

4.     latency reduction for QoS

5.     edge analytics/stream minig

Fog computing will gain prominence as IoT usage grows. With inexpensive low power processing and storage becoming available we can expect computation to move even more closer to the edge and become ingrained in the same devices that are generating data, creating even greater possibilities for inter-device intelligence and interactions.