Packt+ | Advance your knowledge in tech

You're reading from Securing Hadoop Implement robust end-to-end security for your Hadoop ecosystem

Product type Paperback

Published in Nov 2013

Publisher Packt

ISBN-13 9781783285259

Length 116 pages

Edition 1st Edition

Languages

Java

Tools

Hadoop

Concepts

Data Processing

Author (1):

Sudheesh Narayan

View More author details

Table of Contents (15) Chapters

Securing Hadoop

Credits

About the Author

About the Reviewers

www.PacktPub.com

Preface

1. Hadoop Security Overview FREE CHAPTER

2. Hadoop Security Design

3. Setting Up a Secured Hadoop Cluster

4. Securing the Hadoop Ecosystem

5. Integrating Hadoop with Enterprise Security Systems

6. Securing Sensitive Data in Hadoop

7. Security Event and Audit Logging in Hadoop

Solutions Available for Securing Hadoop

Index

A

Access control / Key security considerations
access control list (ACL) / Setting up the administrator principal for KDC
Add/synch feature / HUE
appdefaults property / Configuring the Key Distribution Center
appender for security logging (RFAS) / Configuring Hadoop audit logs
Audits and event monitoring / Key security considerations
Authentication / Key security considerations
Authentication Service (AS) / Key Kerberos terminologies
Authorization / Key security considerations

B

banned.users property / Setting up the TaskController class
Big Data security
- reference architecture / Reference architecture for Big Data security
- reference architecture / Reference architecture for Big Data security
Block Access Token / Block Access Token
business intelligence (BI) / Challenges for securing the Hadoop ecosystem

C

capaths property / Configuring the Key Distribution Center
chmod command / The Hadoop default security model without Kerberos
chown command / The Hadoop default security model without Kerberos
Cloudera Distribution of Hadoop (CDH4) / Configuring Hadoop with Kerberos authentication
Cloudera Manager / Automation of a secured Hadoop deployment
- features / Cloudera Manager
Command Line Interface (CLI) / Securing Hive
Common Security Audit logging / Audit logging, security policies, and procedures
core-site.xml file / Implementing data encryption in Hadoop
Corporate Network / Accessing a secured Hadoop cluster from an enterprise network

D

Dataguise (DG) / Dataguise for Hadoop
Data masking and encryption / Key security considerations
DataNode directory / The Hadoop default security model without Kerberos
dbdefaults property / Configuring the Key Distribution Center
dbmodules property / Configuring the Key Distribution Center
Delegation Token / Delegation Token
dfs.block.access.token.enable property / HDFS-related configurations
dfs.datanode.address property / HDFS-related configurations
dfs.datanode.data.dir.perm property / HDFS-related configurations
dfs.datanode.http.address property / HDFS-related configurations
dfs.datanode.kerberos.principal property / HDFS-related configurations
dfs.datanode.keytab.file property / HDFS-related configurations
dfs.hosts property / The Hadoop default security model without Kerberos
dfs.namenode.kerberos.internal.spnego.principal property / HDFS-related configurations
dfs.namenode.kerberos.principal property / HDFS-related configurations
dfs.namenode.keytab.file property / HDFS-related configurations
dfs.secondary.namenode.kerberos.internal.spnego.principal property / HDFS-related configurations
dfs.secondary.namenode.kerberos.principal property / HDFS-related configurations
dfs.secondary.namenode.keytab.file property / HDFS-related configurations
domain_realm property / Configuring the Key Distribution Center

E

eCryptfs / eCryptfs for Hadoop
EIM
- integrating / Integrating Enterprise Identity Management systems
- users credentials, managing / Integrating Enterprise Identity Management systems
- Active Directory-based EIM, integrating with Hadoop ecosystem / Integrating Active-Directory-based EIM with the Hadoop ecosystem
EIM integration
- configuring, with Hadoop / Configuring EIM integration with Hadoop
Enterprise Security Systems / Configuring users for Hadoop
event monitoring, Hadoop cluster
- User login and authorization events / Security Incident and Event Monitoring in a Hadoop Cluster
- HDFS file operation errors / Security Incident and Event Monitoring in a Hadoop Cluster
- Hadoop RPC authorization errors / Security Incident and Event Monitoring in a Hadoop Cluster
- Hadoop RPC authentication errors / Security Incident and Event Monitoring in a Hadoop Cluster
- HDFS-sensitive file download operations / Security Incident and Event Monitoring in a Hadoop Cluster
- MapReduce job events / Security Incident and Event Monitoring in a Hadoop Cluster
- Exception events / Security Incident and Event Monitoring in a Hadoop Cluster
events
- monitoring / Security Incident and Event Monitoring

F

File System Security / OS and filesystem security
Flume / Challenges for securing the Hadoop ecosystem
- securing / Securing Flume
- sources, securing / Securing Flume sources
- channel, securing / Securing a Flume channel
Flume sources
- securing / Securing Flume sources

G

Gateway Server / Accessing a secured Hadoop cluster from an enterprise network
Gazzang zNcrypt / Gazzang zNcrypt

H

Hadoop
- default security model / The Hadoop default security model without Kerberos
- configuring, with Kerberos authentication / Configuring Hadoop with Kerberos authentication
- users, configuring for / Configuring users for Hadoop
- sensitive data, securing in / Securing sensitive data in Hadoop
hadoop.log.dir property / Setting up the TaskController class
hadoop.security.authentication property / HDFS-related configurations
hadoop.security.authorization property / HDFS-related configurations
Hadoop audit logs, configuring
- common properties for rolling file appender / Configuring Hadoop audit logs
- Hadoop RPC event logging / Configuring Hadoop audit logs
- Hadoop File System access audit logging / Configuring Hadoop audit logs
- Hadoop MapReduce audit logging / Configuring Hadoop audit logs
- HBase audit logging / Configuring Hadoop audit logs
- KDC audit logging / Configuring Hadoop audit logs
Hadoop cluster
- setting up, pre-requisites / Prerequisites
- security incident / Security Incident and Event Monitoring in a Hadoop Cluster
- events monitoring / Security Incident and Event Monitoring in a Hadoop Cluster
- Audit Logging, setting up / Setting up audit logging in a secured Hadoop cluster
- Hadoop audit logs, configuring / Configuring Hadoop audit logs
Hadoop configuration, with Kerberos authentication
- about / Configuring Hadoop with Kerberos authentication
- Kerberos client, setting up / Setting up the Kerberos client on all the Hadoop nodes
- Hadoop service principals, setting up / Setting up Hadoop service principals
Hadoop data encryption, options
- Dataguise (DG) / Dataguise for Hadoop
- Gazzang zNcrypt / Gazzang zNcrypt
- eCryptfs / eCryptfs for Hadoop
Hadoop Distributed File System (HDFS) / Setting up Hadoop service principals
Hadoop Ecosystem
- Kerberos, configuring for / Configuring Kerberos for Hadoop ecosystem components
- securing, best practices / Best practices for securing the Hadoop ecosystem components
Hadoop ecosystem
- securing / Why do we need to secure Hadoop?
- securing, challenges / Challenges for securing the Hadoop ecosystem
- Sqoop / Challenges for securing the Hadoop ecosystem
- Flume / Challenges for securing the Hadoop ecosystem
- Sqoop 2, Flume-ng / Challenges for securing the Hadoop ecosystem
- Hive Server 2 / Challenges for securing the Hadoop ecosystem
- Cloudera Sentry / Challenges for securing the Hadoop ecosystem
- Hortonworks Knox Gateway / Challenges for securing the Hadoop ecosystem
- Project Rhino / Challenges for securing the Hadoop ecosystem
- key security considerations / Key security considerations
- securing, Project Rhino / Securing the Hadoop ecosystem with Project Rhino
Hadoop Kerberos security implementation
- about / Hadoop Kerberos security implementation
- user-level access controls / User-level access controls
- service-level access controls / Service-level access controls, Block Access Token
- impersonation / Service-level access controls
- Self-Served / Service-level access controls
- Secure IPC / Service-level access controls
- user authentication / User and service authentication
- Delegation Token authentication / Delegation Token
- Job Token / Job Token
- Block Access Token / Block Access Token
Hadoop service principals
- setting up / Setting up Hadoop service principals, Creating a keytab file for the Hadoop services , Distributing the keytab file for all the slaves, HDFS-related configurations, MRV1-related configurations, MRV2-related configurations, Setting up the TaskController class
- keytab file, creating / Creating a keytab file for the Hadoop services
- keytab file, distributing / Distributing the keytab file for all the slaves
- Hadoop configuration files, setting up / Setting up Hadoop configuration files
- HDFS-related configurations / HDFS-related configurations
- MRV1-related configurations / MRV1-related configurations
- MRV2-related configurations / MRV2-related configurations
- secured DataNode, setting up / Setting up secured DataNode
- TaskController class, setting up / Setting up the TaskController class
Hadoop sink
- securing / Securing Hadoop sink
HBase
- securing / Securing HBase
Hive
- securing / Securing Hive
- securing, Sentry used / Securing Hive using Sentry
Hive Server 2 / Challenges for securing the Hadoop ecosystem
host-based intrusion detection system (HIDS) / The Security Incident and Event Monitoring (SIEM) system
HttpFS
- about / HttpFS
- using / HttpFS
HTTP Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO) / Securing Oozie
HUE
- about / HUE
- limitations / HUE

I

Identity and Access Management (IDAM) / Challenges for securing the Hadoop ecosystem
Impala / Securing Hive using Sentry
Infrastructure security / Key security considerations
Intel Distribution, of Apache Hadoop
- features / Hadoop distribution with enhanced security support
Intel Manager / Automation of a secured Hadoop deployment

J

Java Authentication and Authorization Service (JAAS) / Securing HBase
Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy File / Supporting AES-256 encryption for a Kerberos ticket
Job Token / Job Token
Jsvc / Setting up secured DataNode

K

kadmin.local utility / Key Kerberos terminologies
kadmind daemons / Key Kerberos terminologies
kadmin utility / Key Kerberos terminologies
kdb5_util utility / Key Kerberos terminologies
KDC
- installing / Installing the Key Distribution Center
- configuring / Configuring the Key Distribution Center
- database, establishing / Establishing the KDC database
- administrator principal, setting up / Setting up the administrator principal for KDC
- Kerberos daemons, starting up / Starting the Kerberos daemons
- Kerberos administrator, setting up / Setting up the first Kerberos administrator
- user(service) principles, adding / Adding the user or service principals
- LDAP, configuring as Kerberos database / Configuring LDAP as the Kerberos database
- AES-256 encryption, supporting Kerberos ticket / Supporting AES-256 encryption for a Kerberos ticket
kdcdefaults property / Configuring the Key Distribution Center
Kerberos
- about / What is Kerberos?
- heads / What is Kerberos?
- terminologies / Key Kerberos terminologies
- krb5kdc daemons / Key Kerberos terminologies
- kadmind daemons / Key Kerberos terminologies
- utilities / Key Kerberos terminologies
- working, diagram / How Kerberos works?
- working, steps / How Kerberos works?
- advantages / Kerberos advantages
- setting up / Setting up Kerberos
Kerberos, setting up
- diagram / Setting up Kerberos
- Kerberos utilities diagram / Setting up Kerberos
- KDC, installing / Installing the Key Distribution Center
Kerberos, terminologies
- Authentication Service (AS) / Key Kerberos terminologies
- Ticket Granting Service (TGS) / Key Kerberos terminologies
- realm / Key Kerberos terminologies
Kerberos, utilities
- kadmin / Key Kerberos terminologies
- kadmin.local / Key Kerberos terminologies
- kinit / Key Kerberos terminologies
- klist / Key Kerberos terminologies
- ktutil / Key Kerberos terminologies
- kdb5_util / Key Kerberos terminologies
Kerberos configuration, for Hadoop Ecosystem
- Hive, securing / Securing Hive
- Oozie, securing / Securing Oozie
- Flume, securing / Securing Flume
- HBase, securing / Securing HBase
- Sqoop, securing / Securing Sqoop
- Pig, securing / Securing Pig
Key Distribution Center (KDC) / What is Kerberos?
key security considerations, Hadoop ecosystem
- authentication / Key security considerations
- authorization / Key security considerations
- access control / Key security considerations
- Data masking and encryption / Key security considerations
- Network perimeter security / Key security considerations
- system security / Key security considerations
- infrastructure security / Key security considerations
- audits and event monitoring / Key security considerations
keystore-password property / Securing Flume sources
keystore-type property / Securing Flume sources
keystore property / Securing Flume sources
keytab file / Distributing the keytab file for all the slaves
kinit command / Setting up the first Kerberos administrator
kinit utility / Key Kerberos terminologies
klist utility / Key Kerberos terminologies
Knox Gateway Server
- about / Knox Gateway Server
- diagram / Knox Gateway Server
krb5kdc daemons / Key Kerberos terminologies
ktutil utility / Key Kerberos terminologies

L

LDAP Synchronization Connector (LSC) / Configuring EIM integration with Hadoop
libdefaults property / Configuring the Key Distribution Center
logging property / Configuring the Key Distribution Center

M

mapred-site.xml file / Implementing data encryption in Hadoop
mapred.task.tracker.task-controller property / MRV1-related configurations
mapreduce.jobhistory.keytab property / MRV2-related configurations
mapreduce.jobhistory.principal property / MRV2-related configurations
mapreduce.jobtracker.kerberos.principal property / MRV1-related configurations
mapreduce.jobtracker.keytab.file property / MRV1-related configurations
mapreduce.tasktracker.group property / MRV1-related configurations, Setting up the TaskController class
mapreduce.tasktracker.kerberos.principal property / MRV1-related configurations
mapreduce.tasktracker.keytab.file property / MRV1-related configurations
Master / Securing HBase
min.user.id property / Setting up the TaskController class

N

Network perimeter security / Key security considerations
network perimeter security / Network perimeter security
Null appenders / Setting up audit logging in a secured Hadoop cluster

O

Oozie
- securing / Securing Oozie
Operating System (OS) / Integrating Enterprise Identity Management systems

P

Pig
- securing / Securing Pig
principals / Key Kerberos terminologies
Project Rhino / Implementing data encryption in Hadoop
- used, for Hadoop ecosystem security / Securing the Hadoop ecosystem with Project Rhino

R

realm / Key Kerberos terminologies
realms property / Configuring the Key Distribution Center
reference architecture
- used, for security technologies mapping / Mapping of security technologies with the reference architecture
reference architecture, for Big Data security / Reference architecture for Big Data security
Region / Securing HBase
RegionServer / Securing HBase
role-based access controls (RBACs) / Zettaset

S

secured Hadoop cluster
- accessing, in enterprise network / Accessing a secured Hadoop cluster from an enterprise network
- Corporate Network / Accessing a secured Hadoop cluster from an enterprise network
- Gateway Server / Accessing a secured Hadoop cluster from an enterprise network
- HttpFS / HttpFS
- HUE / HUE
- Knox Gateway Server / Knox Gateway Server
secured Hadoop cluster deployment automation
- Cloudera Manager tool / Cloudera Manager
- Zettaset tool / Zettaset
secured Hadoop deployment
- automating / Automation of a secured Hadoop deployment
securing insights approach, Hadoop
- data in motion, securing / Securing data in motion
- data at rest, securing / Securing data at rest
- data encryption, implementing / Implementing data encryption in Hadoop
security incident / Security Incident and Event Monitoring
security incident, Hadoop cluster / Security Incident and Event Monitoring in a Hadoop Cluster
security technologies mapping,reference architecture used
- section diagram / Mapping of security technologies with the reference architecture
- infrastructure security / Infrastructure security
- File System Security / OS and filesystem security
- application security / Application security
- network perimeter security / Network perimeter security
- data masking / Data masking and encryption
- encryption / Data masking and encryption
- authentication / Authentication and authorization
- authorization / Authentication and authorization
- audit logging / Audit logging, security policies, and procedures
- security policies / Audit logging, security policies, and procedures
- event Monitoring / Security Incident and Event Monitoring
- Security Incident / Security Incident and Event Monitoring
sensitive data, securing in Hadoop
- categories / Securing sensitive data in Hadoop
- key requirements / Securing sensitive data in Hadoop
- securing insights approach / Approach for securing insights in Hadoop
Sentry
- used, for Hive security / Securing Hive using Sentry
service-level access controls
- about / Service-level access controls
- scalable authentication / Service-level access controls
SIEM system
- Log and event collecting agents / The Security Incident and Event Monitoring (SIEM) system
- Event Monitoring Server / The Security Incident and Event Monitoring (SIEM) system
- Event Monitoring and Audit Logging UI / The Security Incident and Event Monitoring (SIEM) system
- block diagram / The Security Incident and Event Monitoring (SIEM) system
Simple Authentication and Security Layer (SASL) / User and service authentication
Sqoop / Challenges for securing the Hadoop ecosystem
- securing / Securing Sqoop
ssl property / Securing Flume sources
System security / Key security considerations

T

TaskController class / Setting up the TaskController class
ticket-granting ticket (TGT) / Securing Pig
Ticket Granting Service (TGS) / Key Kerberos terminologies
Ticket Granting Ticket (TGT) / How Kerberos works?

U

user-level access controls / User-level access controls
users
- configuring, for Hadoop / Configuring users for Hadoop

Y

yarn.nodemanager.container-executor.class property / MRV2-related configurations
yarn.nodemanager.keytab property / MRV2-related configurations
yarn.nodemanager.linux-container-executor.group property / MRV2-related configurations, Setting up the TaskController class
yarn.nodemanager.log-dirs property / Setting up the TaskController class
yarn.nodemanager.principal property / MRV2-related configurations
yarn.resourcemanager.keytab property / MRV2-related configurations
yarn.resourcemanager.principal property / MRV2-related configurations