I’ve been doing some noodling around with Confluent’s Kafka Connect recently, as part of gaining a wider understanding into Kafka. If you’re not familiar with Kafka Connect this page gives a good idea of the thinking behind it. One issue that I hit defeated my Google-fu so I’m recording it here to hopefully help out fellow n00bs. The pipeline that I’d set up looked like this: Eneco’s Twitter Source streaming tweets to a Kafka topic Confluent’s HDFS Sink to stream tweets to HDFS and define Hive table automagically over them It worked great, but only if I didn’t enable the Hive integration part.

Continue Reading

With my new server I bought a UPS, partly just as a Good Thing, but also because I suspect a powercut fried the motherboard on a previous machine that I had, and this baby is too precious to lose ;) The idea is that the UPS will smooth out the power supply to my server, protecting it from surges or temporarily blips in power loss. If there’s a proper power cut, the UPS is connected to my server and can initiate a graceful shutdown instead of system crash.

Continue Reading

Trying to use SparkSQL to read a JSON file, from either pyspark or spark-shell, I got this error: java.io.IOException: No input paths specified in job scala> sqlContext.read.json("/u02/custom/twitter/twitter.json") java.io.IOException: No input paths specified in job at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:202) Despite the reference articles that I found using this local path syntax (/u02/custom/twitter/twitter.json), it turned out that I needed to prefix it with file://: scala> sqlContext.read.json("file:///u02/custom/twitter/twitter.json") res3: org.apache.spark.sql.DataFrame = [@timestamp: string, @version: string, contributors: string, coordinates: string, created_at: string, entities: struct<hashtags:array<struct<indices:array<bigint>,text:string>>,media:array<struct<display_url:string,expanded_url:string,id:bigint,id_str:string,indices:array<bigint>,media_url:string,media_url_https:string,sizes:struct<large:struct<h:bigint,resize:string,w:bigint>,medium:struct<h:bigint,resize:string,w:bigint>,small:struct<h:bigint,resize:string,w:bigint>,thumb:struct<h:bigint,resize:string,w:bigint>>,source_status_id:bigint,source_status_id_str:string,source_user_id:bigint,source_user_id_str:string,type:string,url:string>>,symbols:array<struct<indices:array<bigint>,text:string>>,urls:array<struct<display_url:string,expanded_url:string.

Continue Reading

TL;DR When defining networking on Proxmox 4 LXC containers, use an appropriate CIDR suffix (e.g. 24) - don’t use 32! On my Proxmox 4 server I’m running a whole load of lovely LXC containers. Unfortunately, I had trouble connecting to them. From a client machine, I got the error ssh_exchange_identification: read: Connection reset by peer On the server I was connecting to (which I could get a console for through the Proxmox GUI, or a session on using pct enter from the Proxmox host) I ran a SSHD process with debug to see what was happening:

Continue Reading

(Ref) The bit that caught me out was this kept failing with Error: Password not present and a Python stack trace that ended with subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/78-hue-HUE_SERVER/altscript.sh sec-1-secret_key' returned non-zero exit status 1 The answer (it seems) is to ensure that HUE_SECRET_KEY is set (to any value!) Launch shell: export HUE_SECRET_KEY=foobar /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue shell Reset password for hue, activate account and make it superuser from django.contrib.auth.models import User user = User.objects.get(username='hue') user.

Continue Reading

Vanilla download of Apache Drill 1.6, attempting to follow the Followed the Drill in 10 Minutes tutorial - but kept just getting the error No current connection. Here’s an example: [oracle@bigdatalite apache-drill-1.6.0]$ ./bin/drill-embedded Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 com.fasterxml.jackson.databind.JavaType.isReferenceType()Z apache drill 1.6.0 "the only truly happy people are children, the creative minority and drill users" 0: jdbc:drill:zk=local> SELECT version FROM sys.version; No current connection 0: jdbc:drill:zk=local> Whether SELECT version FROM sys.

Continue Reading

I wasted literally two hours on this one, so putting down a note to hopefully help future Googlers. Symptom Here’s all the various errors that I got in the hive-server2.log during my attempts to get a CREATE EXTERNABLE TABLE to work against a MongoDB table in Hive: Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.io.BSONWritable Caused by: java.lang.ClassNotFoundException: com.mongodb.util.JSON Caused by: java.lang.ClassNotFoundException: org.bson.conversions.Bson Caused by: java.lang.ClassNotFoundException: org.bson.io.OutputBuffer Whilst Hive would throw errors along the lines of:

Continue Reading

Installing CDH 5.7 on Linux Containers (LXC) hosted on Proxmox 4. Everything was going well until Cluster Setup, and which point it failed on Start YARN (MR2 included) Completed only 0/1 steps. First failure: Failed to execute command Start on service YARN (MR2 Included) Log /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-cdh57-01-node-02.moffatt.me.log.out showed: org.apache.hadoop.service.AbstractService: Service containers-monitor failed in state INITED; cause: java.lang.NumberFormatException: For input string: "18446744073709550364" java.lang.NumberFormatException: For input string: "18446744073709550364" Looking down the stack trace, this came from org.

Continue Reading

I’ve got an EdgeRouter LITE (ERL) which I used as my home router until a powercut fried it a while ago (looks like I’m not the only one to have this issue). The symptoms were it powering on but not giving any DHCP addresses, or after a factory reset responding on the default IP of 192.168.1.1. It was a real shame, because it had been a great bit of kit up until then.

Continue Reading

(Previously, previously, previously) Since Proxmox 4 has a recent Linux kernel and mainline one at that, it means that Docker can be run on it. I’ve yet to really dig into Docker and work out when it makes sense in place of Linux Containers (LXC), so this is going to be a learning experience for me. To install Docker, add Backports repo to apt: root@proxmox01:~# cat /etc/apt/sources.list.d/backports.list deb http://ftp.debian.org/debian jessie-backports main And then install:

Continue Reading

(Previously, previously) I’ve got a bunch of existing VirtualBox and VMWare VMs that I want to run on Proxmox. Eventually I’ll migrate them to containers, but for the time being run them as “fat” VMs using Proxmox’s KVM virtualisation. After copying the OVA files that I had to the server, I uncompressed them: root@proxmox01:/data04/vms/bdl44-biwa# cd ../bdl44 root@proxmox01:/data04/vms/bdl44# ll total 27249328 -rw------- 1 root root 27903306752 Jun 1 10:14 BigDataLite440.ova root@proxmox01:/data04/vms/bdl44# tar -xf BigDataLite440.

Continue Reading

(Previously) With my server in place, I ran a memtest on it … which with 128G took a while ;) And then installed Proxmox 4, using a bootable USB that I’d created on my Mac from the ISO downloaded from Proxmox’s website. To create the bootable USB, create the img file: hdiutil convert -format UDRW -o target.img source.iso and then burn it to USB: sudo dd if=target.img of=/dev/rdiskN bs=1m Replace N with the correct device based on diskutil list output.

Continue Reading

After a long and painful delivery, I’m delighted to announce the arrival of a new addition to my household … : This custom-build from Scan 3XS is sat in my study quietly humming away. I’m going to use it for hosting VMs for R&D on OBIEE, Big Data Lite, Elastic, InfluxDB, Kafka, etc. I’ll blog various installations that I’ve done on it as a reference for myself, and anyone else interested.

Continue Reading

Oracle’s excellent Big Data Lite VM has been updated, to version 4.5. Download it here Changes: CDH 5.5 -> 5.7 Big Data Spatial and Graph 1.1 -> 1.2 Big Data Discovery 1.1 -> 1.2 Oracle Big Data Connectors 4.4 -> 4.5 Oracle NoSQL 3.5 -> 4.0 GoldenGate 12.2.0.1 -> 12.2.0.1.1

Continue Reading

I’ve been spending some interesting hours digging into OBIEE 12c recently, with some interesting blog posts to show for it. Some of it is just curiosities discovered along the way, but the real meaty stuff is the in the RESTful APIs - lots of potential here for cool integrations I think… Lifting the Lid on OBIEE 12c Web Services - Part 1 Lifting the Lid on OBIEE 12c Web Services - Part 2 Extended Subject Areas (XSA) and the Data Set Service Changes in BI Server Cache Behaviour in OBIEE 12c : OBIS_REFRESH_CACHE Dynamic Naming of OBIEE 12c Service Instance Exports OBIEE 12c - “Add Data Source” in Answers (Photo credit: https://unsplash.

Continue Reading

Presentation Services can provide some very detailed logs, useful for troubleshooting, performance tracing, and general poking around. See here for details. There’s no bi-init.sh in 12c, so need to set up the LD_LIBRARY_PATH ourselves: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/oracle/biee/bi/bifoundation/web/bin/:/app/oracle/biee/bi/lib/:/app/oracle/biee/lib/:/app/oracle/biee/bi/bifoundation/odbc/lib/ Run sawserver with flag to list all log sources /app/oracle/biee/bi/bifoundation/web/bin/sawserver -logsources > saw_logsources_12.2.1.txt Full list: https://gist.github.com/rmoff/e3be9009da6130839c71181cb58509a0

Continue Reading

In OBIEE 12c data-model-cmd is a wrapper for some java code which ultimately calls an internal RESTful web service in OBIEE 12c, bi-lcm. We saw in the previous post how these internal web services can be opened up slightly, and we’re going to do the same again here. Which means, time for the same caveat: None of these Web Services are documented, and they should therefore be assumed to be completely unsupported by Oracle.

Continue Reading

exportServiceInstance will export the RPD, Presentation Catalog, and Security model (application roles & policies etc – but not WLS LDAP) into a single .bar file, from which they can be imported to another environment, or restored to the same one at a later date (e.g. for backup/restore). To run exportServiceInstance you need to launch WLST first. The following demonstrates how to call it, and embeds the current timestamp & machine details in the backup (useful info, and also makes the backup name unique each time).

Continue Reading

So this had me scratching my head for a good hour today. Comparing SampleApp v511 against a vanilla OBIEE 12c install I’d done, one had “Add Data Source” as an option in Answers, the other didn’t. The strange thing was that the option wasn’t there in SampleApp – and usually that has all the bells and whistles enabled. After checking and re-checking the Manage Privileges option, and even the Application Policy grants, and the manual, I hit MoS - and turned up Doc ID 2093886.

Continue Reading

I had the pleasure of not one but two fry-ups in York, UK last weekend. The first was courtesy of Bill’s Restaurant Overall, pretty good, and I’ve had much worse. All the ingredients seemed decent. The black pudding was overcooked and almost biscuit-like, but that’s my only serious grumble. The bacon was cooked well. That black pudding, beans and the mashed/friend potato thing were each extra charges annoyed me. Particularly with a hangover, I just want to be able to order a full english, without playing Mastermind to work out what’s in or not.

Continue Reading