Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Multi-Tenant Apache Kafka for Hops: Kafka Topic-Based Multi-Tenancy and ACL- Based Authorization for Hops
KTH, School of Information and Communication Technology (ICT).
2016 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE creditsStudent thesis
Abstract [en]

Apache Kafka is a distributed, high throughput and fault-tolerant publish/subscribe messaging system in the Hadoop ecosystem. It is used as a distributed data streaming and processing platform. Kafka topics are the units of message feeds in the Kafka cluster. Kafka producer publishes messages into these topics and a Kafka consumer subscribes to topics to pull those messages. With the increased usage of Kafka in the data infrastructure of many companies, there are many Kafka clients that publish and consume messages to/from the Kafka topics. In fact, these client operations can be malicious. To mitigate this risk, clients must authenticate themselves and their operation must be authorized before they can access to a given topic. Nowadays, Kafka ships with a pluggable Authorizer interface to implement access control list (ACL) based authorization for client operation. Kafka users can implement the interface differently to satisfy their security requirements. SimpleACLAuthorizer is the out-of-box implementation of the interface and uses a Zookeeper for ACLs storage.HopsWorks, based on Hops a next generation Hadoop distribution, provides support for project-based multi-tenancy, where projects are fully isolated at the level of the Hadoop Filesystem and YARN. In this project, we added Kafka topicbased multi-tenancy in Hops projects. Kafka topic is created from inside Hops project and persisted both at the Zookeeper and the NDBCluster. Persisting a topic into a database enabled us for topic sharing across projects. ACLs are added to Kafka topics and are persisted only into the database. Client access to Kafka topics is authorized based on these ACLs. ACLs are added, updated, listed and/or removed from the HopsWorks WebUI. HopsACLAuthorizer, a Hops implementation of the Authorizer interface, authorizes Kafka client operations using the ACLs in the database. The Apache Avro schema registry for topics enabled the producer and consumer to better integrate by transferring a preestablished message format. The result of this project is the first Hadoop distribution that supports Kafka multi-tenancy.

Place, publisher, year, edition, pages
2016. , p. 52
Series
TRITA-ICT-EX ; 2016:120
Keywords [en]
Hadoop, Kafka, Hops, HopsWorks, Multi-Tenancy, Kafka Topics, Schema Registry, Messaging Systems, ACL Authorization
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-206086OAI: oai:DiVA.org:kth-206086DiVA, id: diva2:1091136
Subject / course
Computer Science
Educational program
Master of Science - Software Engineering of Distributed Systems
Examiners
Available from: 2017-04-26 Created: 2017-04-26 Last updated: 2018-01-13Bibliographically approved

Open Access in DiVA

fulltext(1185 kB)458 downloads
File information
File name FULLTEXT01.pdfFile size 1185 kBChecksum SHA-512
6bbed33c0c75803f04f383d9983cd4afc856dd1d678a25150de22ca6403fdbf0d1a0a9ce147f3d4397899dfd76cf7ec76e8aed362ed8ef10e33cba3993237014
Type fulltextMimetype application/pdf

By organisation
School of Information and Communication Technology (ICT)
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 458 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 762 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf