How to use Amazon S3 from KNIME

This post follows on from my S3 post usng Node-RED so you can read that for a brief into to S3. Using S3 with KNIME is extremely easy as there is already an S3 node available to install.

Amazon S3 Connection

The only node that needs installing is the Amazon S3 Connection.

KNIME Amazon Cloud Connections

All the other nodes required to upload, download or list remote files are already installed in the IO section of the Node Repository in KNIME as shown here.

Amazon S3 connection in the IO section

A simple workflow to upload a file and list remote files in the S3 bucket looks like this.

S3 Credentials

The S3 connection node is easy to set up and I recommend using the standard AWS credentials file so your do not save your id and key in the KNIME workflow. It’s clearly documented on the AWS site.

The only trick is to create folder starting with a dot (.) is not possible on Windows using Windows Explorer. To create the folder you must open a command line and use the mkdir command.

C:\Users\username>mkdir .aws

S3 Connection Node

Configure the node to use your credentials file and select your AWS region. Test the connection before continuing to make sure everything works as expected.

Upload to S3

Simply choose the target directory on S3 you wish to upload your files to, in this case I’m uploading to a sub directory called nodered. As well as the S3 connection this node expects a list of URIs as shown in the initial flow overview. This is standard KNIME and uses a List Files node followed by a String to URI node.

List S3 Files

As with uploading this node asks for the S3 directory to list and by default will list all the files in you bucket.

Here the remote files are displayed.

Leave a comment

Your email address will not be published.