Use Apache Zeppelin as an interactive notebook that enables interactive data exploration.
Apache Zeppelin is a web-based notebook for data analysis, visualisation and reporting. Zeppelin lets you perform data analysis interactively and view the outcome of your analysis visually. It supports the Scala functional programming language with Spark by default. If you have used Jupyter Notebook (previously known as IPython Notebook) or Databricks Cloud before, you will find Zeppelin familiar.
Create AWS EMR cluster with Zeppelin.
Once you have created AWS EMR cluster with zeppelin ,you need to make configuration changes to use other interpreters like JDBC,PIG,Python,HBASE, livy for sparkR ,etc.These interpreters helps you to access different data sources in you environment.
Prerequisite:
1)Install maven
- sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
- sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
- sudo yum install -y apache-maven
- mvn --version
2)Install git(if not available)
sudo yum install git.
3)Install livy server .An Open Source REST Service for Apache Spark (Apache License)
git clone https://github.com/cloudera/livy.git
cd livy
mvn package -DskipTests
sudo vi ~/.bashrc
add below in path
export SPARK_HOME=/usr/lib/spark
export HADOOP_CONF_DIR=/etc/hadoop/conf
source ~/.bashrc nohup ./bin/livy-server
Check if the livy has started by copying url to browser:http://ip-xx-xx-xx-xx.ap-southeast-2.compute.internal:8998/ui
Livy server in interpreter section of zeppelin
To add JDBC Interpreter into the zeppelin notebooks to access phoenix, aurora,redshift etc.
- Install all community managed interpreters
- ./bin/install-interpreter.sh --all
- Install specific interpreters
- ./bin/install-interpreter.sh --name md,shell,jdbc,python
- You can get full list of community managed interpreters by running
- ./bin/install-interpreter.sh --list
Then add the same on interpreter section of zeppelin .Please go through https://zeppelin.apache.org/docs/0.7.1/interpreter/jdbc.html to configure redshift,aurora,hive,etc to jdbc.
restart the zeppelin after adding new interpreter each time: sudo /usr/lib/zeppelin/bin/zeppelin-daemon.sh restart.
To add LDAP configurations through shiro.
Apache Shiro is a powerful and easy-to-use Java security framework that performs authentication, authorization, cryptography, and session management. In this documentation, we will explain step by step how Shiro works for Zeppelin notebook authentication.
When you connect to Apache Zeppelin, you will be asked to enter your credentials. Once you logged in, then you have access to all notes including other user's notes.
1. Secure the HTTP channel
To secure the HTTP channel, you have to change both anon and authc settings in conf/shiro.ini. In here, anonmeans "the access is anonymous" and authc means "formed auth security".
The default status of them is
/** = anon
#/** = authc
Deactivate the line "/** = anon" and activate the line "/** = authc" in conf/shiro.ini file.
#/** = anon
/** = authc
For the further information about shiro.ini file format, please refer to Shiro Configuration.
2. Secure the Websocket channel
Set to property zeppelin.anonymous.allowed to false in conf/zeppelin-site.xml. If you don't have this file yet, just copy conf/zeppelin-site.xml.template to conf/zeppelin-site.xml.
3. Start Zeppelin
bin/zeppelin-daemon.sh start (or restart).
4.Groups and permissions for LDAP
In case you want to leverage user groups and permissions, use one of the following configuration for LDAP or AD under [main] segment in shiro.ini
eg of shiro.ini
[main] ### A sample for configuring Active Directory Realm
activeDirectoryRealm = org.apache.zeppelin.realm.ActiveDirectoryGroupRealm
#activeDirectoryRealm.systemUsername = userNameA
#activeDirectoryRealm = org.apache.zeppelin.server.ActiveDirectoryGroupRealm activeDirectoryRealm.systemUsername = cn=svc.aws.abc,ou=ServiceAccounts,ou=ABC,Users,dc=ABC,dc=abccompany,dc=com,dc=xy
activeDirectoryRealm.systemPassword = password!
activeDirectoryRealm.searchBase = dc=xyz,dc=abccompany,dc=com,dc=xy
activeDirectoryRealm.url = ldap://yourldapport:389
activeDirectoryRealm.groupRolesMap = "cn=Right-Usr-AP-Science.Zeppelin.Developer-U-GS,ou=Rights,ou=ABCGroups,dc=xyz,dc=abccompany,DC=com,DC=xy":"developers"
activeDirectoryRealm.hadoopSecurityCredentialPath = jceks://user/zeppelin/conf/zeppelin.jceks
[roles]
#role1 = *
#role2 = *
#role3 = *
admin = *
developers = *
make sure LDAP and other configurations are correct. Follow this path for details https://zeppelin.apache.org/docs/0.7.1/security/shiroauthentication.html
restart zeppelin.
Externalize notebooks to S3
In Zeppelin there are two option for storage Notebook, by default the notebook is storage in the notebook folder in your local File System and the second option is S3.
Notebook Storage in S3
you need the following folder structure on S3
bucket_name/
username/
notebook/
set the enviroment variable in the file zeppelin-env.sh:
export ZEPPELIN_NOTEBOOK_S3_BUCKET = bucket_name
export ZEPPELIN_NOTEBOOK_S3_USER = username
eg
export ZEPPELIN_NOTEBOOK_S3_BUCKET=abc-science-data-dev
export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
export ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
in the file zeppelin-site.xml uncommet and complete the next property:
<property>
<name>zeppelin.notebook.s3.user</name>
<value>username</value>
<description>user name for s3 folder structure</description>
</property>
<property>
<name>zeppelin.notebook.s3.bucket</name>
<value>bucket_name</value>
<description>bucket name for notebook storage</description>
</property>
uncomment the next property for use S3NotebookRepo class:
<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.S3NotebookRepo</value>
<description>notebook persistence layer implementation</description>
</property>
comment the next property:
<property>
<name>zeppelin.notebook.storage</name>
<value>org.apache.zeppelin.notebook.repo.VFSNotebookRepo</value>
<description>notebook persistence layer implementation</description>
</property>
restart zeppelin
