Uploading data via the EDI-Net API

Getting data into EDI-Net is easy… if you know how.

I have previously written a more general overview of the data transfer process. If you haven’t read that then I recommend looking it over before reading any further.

This post is aimed at a technical audience. I am specifically going to talk about the technical steps necessary to automate the uploading of data into the EDI-Net dashboard. As such it may include technical jargon which needs explanation. Please do reply with any issues of clarification and I will update the post accordingly. In this way, I am hoping we will be able to document the process as clearly and unambiguously as possible.

The EDI-Net API

The EDI-Net system exposes an API over HTTPS through which data files can be uploaded automatically. The EDI-Net API is extensive and allows for many interactions with the EDI-Net system. However, for the purposes of this post we will focus on the facility to upload data files.

Supported data file formats

The only way to push data into EDI-Net is to upload data files. Our approach is wherever possible to take your data in their native format or in the format in which your system most easily produces data files. So we maintain a growing set of data adapters which contain the logic needed to extract data from whatever file format you can most easily provide.

A list of supported file formats can be viewed in the EDI-Net dashboard. Right now the list is short as most of our users have APIs from which we “pull” their data.

The list can also be grabbed as a json file from https://dashboard.edi-net.eu/api/dataFormats.

{
  "data-formats": [
    {"label": "default", "id": "default", "description": "default file format"},
    {"label": "catalunya", "id": "catalunya", "description": "file format provided by CIMNE"}, 
    {"label": "EDF", "id": "EDF", "description": "file format provided by EDF energy"}
  ]
}

We don’t recommend using any of these formats unless you happen to use the same systems as our existing users. The default format is not fixed yet, it is under development and will be informed by experience. This will be for users who need to create a new file export facility for their system.

So, if your file format is not currently supported then simply send us an example file and we will let you know if we can add support for your data. We tend to prefer JSON or CSV if possible but any valid XML, JSON or
CSV format is good and whatever your format, we will do our best to work with it. Once we have an adapter in place, it will appear as an option in the interface and in the json file above. It will be given a unique identifier which you will need in order to upload a file.

Obtaining a security token

Most requests must be authorised with a security token in the request ‘Authorization’ header.

Authorization: b156abe2ef124c74ea0eb12dfd84923858ee545e

A security token is associated with a server ‘session’ and can be obtained by making a POST request to: https://dashboard.edi-net.eu/sessions

The request must include a JSON payload containing a valid username and password like so:

{
  "user": {
    "username": "my-username", 
    "password": "my-password"
  }
}

The server will respond either with an error (HTTP 400 for badly formatted data, HTTP 401 for invalid credentials) or, if the credentials are correctly formatted and valid, with an HTTP 200 code and a JSON
formatted response like this:

{
  "session": {
    "token": "444de1cc35526cb4de32c4c0e92de95a41c9f2ac",
    "user_id": "my-username"
  }
}

The token can be extracted from this response and used in future requests by including it in the Authorization header as described above.

Uploading data files—an overview

To upload a file send an HTTPS POST request to: https://dashboard.edi-net.eu/dataFiles

The request should include an authentication token in the header as described above.

The request should also have the following data attached as Content-Type: multipart/form-data;

  • dataFile[format] – a unique identifier representing the format of the attached data file
  • dataFile[organisation] – a unique identifier representing the organisation to which the file should be attached
  • dataFile[file] – the actual file itself

The return value for a successful file upload is a simple JSON document:

{
  "data-file": {
    "created_at": "2018-01-30T13:48:36.841499",
    "url": "/dataFiles/25721/download",
    "imported": false,
    "updated_at": null,
    "filename": "data-file.csv",
    "error": false,
    "id": 2357211,
    "error_message": null,
    "format": "my-format",
    "organisation": "my-organisation"
  }
}

So that’s it. Once you have a successful file upload the file will be placed in a queue for importing into the system and once imported the data will be available to the EDI-Net system.


Some example API usage with ‘cURL’

Many tools can make HTTPS requests. The following examples use cURL, which comes with most Unix-based systems like Linux or Mac OS X, and you can download cURL for Windows here: http://www.confusedbycode.com/curl/

These commands should be entered as a single line of code. For completeness the actual HTTP message is shown for all examples:

1. Obtain a security token:

This command makes the request for a session.

curl -H "Content-Type: application/json" -d "{\"user\":{\"username\":\"your-username\",\"password\":\"your-password\"}}" -k https://dashboard.edi-net.eu/sessions

Notice how we escape the quotation marks in the passed data and include a content type header in the
request.

The request generated by curl looks something like this:

POST /sessions HTTP/1.1
User-Agent: curl/7.35.0
Host: dashboard.edi-net.eu
Accept: */*
Content-Type: application/json
Content-Length: 60

{"user":{"username":"my-username","password":"my-password"}}

You will receive a response similar to this:

HTTP/1.1 200 OK.
Content-Length: 94
Content-Type: application/json; charset=UTF-8
Date: Thu, 01 Feb 2018 11:33:21 GMT
Server: nginx/1.2.1

{
  "session": {
    "user_id": "your-username",
    "token": "5565db93929644766beae7f31d524ee6fca0af18"
  }
}

The token can be extracted from the response and used for the upload request.

2. Use the security token to upload a file:

Here we are adding the “Authorization” header and attaching the file and file meta data to the request.

curl -H "Authorization: 5565db93929644766beae7f31d524ee6fca0af18" --form dataFile[file]=@data_filename_and_extension --form dataFile[organisation]="my-organisation" --form dataFile[format]="my-format" -k https://dashboard.edi-net.eu/dataFiles

The http request generated by curl looks something like this:

POST /dataFiles HTTP/1.1
User-Agent: curl/7.21.1 (i686-pc-mingw32) libcurl/7.21.1 OpenSSL/0.9.8r zlib/1.2.3
Host: dashboard.edi-net.eu
Accept: */*
Authorization: 5565db93929644766beae7f31d524ee6fca0af18
Content-Length: 201
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------49cf25
bae2d7

You will receive a response similar to this:

HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Server: nginx/1.2.1
Date: Tue, 04 Nov 2014 12:47:46 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 44
Connection: keep-alive

{
  "OK": true,
  "filename": "data_filename_and_extension"
}


A complete (bash) script for automating the process

The upload process can easily be automated. In this example we use bash. We define a function in the file login.sh to get a security token and another function in upload.sh to upload a data file. Finally we present a simple script automate.sh for automating a bulk upload.

login.sh

This can be called with arguments specifying the server, the username and the password. The fourth argument will be the name of a variable which will be used to store the token on success.

# !/bin/bash
# This function makes a call to the EDI-Net API to create a session
# It passes in the username and password and returns the token to be used in subsequent requests

function login() {
  if [ $# -ne 4 ]; then
    echo $0: usage: login server username password result
    exit 1
  fi
  local server=$1;
  local username=$2;
  local password=$3;
  local __token=$4;
  local tokenurl="http://$server/sessions"
  echo "requesting a token from $tokenurl";
  if __token_response=$(curl -sS -H "Content-Type: application/json" -d "{\"user\":{\"username\":\"$username\",\"password\":\"$password\"}}" -k $tokenurl); then
    if [[ "$__token_response" =~ \"token\":\ \"([A-Za-z0-9]+)\" ]]; then
      eval "$__token=${BASH_REMATCH[1]}";
    else
      echo "$__token_response";
      exit;
    fi
  else
    echo "Could not get token";
    exit;
  fi
}

upload.sh

Another function is defined in “upload.sh” to upload a data file. This function takes five arguments. We
pass it the server, the filename, the organisation to which the file is to be uploaded, the file format and
the security token.

# !/bin/bash
# This function makes a call to the EDI-Net API to POST a data file
# It also passes the organisation identifier and a file format identifier
# Supported file formats can be viewed in the EDI-Net dashboard or are available on request
# If your file format is not supported then send us an example file and we will develop a data adapter for you

function upload() {
  if [ $# -ne 5 ]; then
    echo $0: usage: upload server filename format organisation token
    exit 1
  fi
  local __server=$1;
  local __upload_file=$2;
  local __format=$3;
  local __organisation=$4;
  local __token=$5;
  local uploadurl="http://$__server/dataFiles";
  echo "uploading $__upload_file...";
  if __upload_response=$(curl -H "Authorization: $__token" --form dataFile[file]=@"$__upload_file" --form dataFile[organisation]="$__organisation" --form dataFile[format]="$__format" -k $uploadurl); then
    if [[ "$__upload_response" =~ \"status\":[[:space:]]\"error\" ]]; then
      printf "\n";
      if [[ "$__upload_response" =~ \"errors\":[[:space:]](\[(.*?)\]) ]]; then
        echo "Error: ${BASH_REMATCH[1]}";
        printf "\n";
        return 2;
      else
        echo "No error message found";
        echo "$__upload_response";
        printf "\n";
        return 3;
      fi
      exit;
    else
      echo "successfully uploaded $__upload_file...";
      echo "$__upload_response";
      printf "\n";
      return 0;
    fi
  else
    echo "Could not upload $__upload_file";
    printf "\n";
    return 1;
    exit;
  fi
}

automate.sh

This script shows how to simply automate the regular upload of data. We define a source and an archive directory. Any files placed in the source directory will be uploaded and moved to the archive directory. Calling this script automatically on a regular basis means that files automatically dropped into the source directory will be uploaded. In this way, it is fairly simple to automate data transfer to the EDI-Net system.

# !/bin/bash
source login.sh;
source upload.sh;
server="dashboard.edi-net.eu";
format="my-format";
organisation="my-organisation";
username="my-username";
password="my-password";
source_directory=~/".edinet/source/";
archive_directory=~/".edinet/archive/"
shopt -s nullglob
cd "$source_directory"
files=(*)
shopt -u nullglob
if [ ${#files[@]} -gt 0 ]; then
  login $server $username $password token;
  for filename in ${files[@]}
  do
    if upload $server "$source_directory$filename" $format $organisation $token; then
      mv "$source_directory$filename" "$archive_directory$filename";
    fi
  done
else
  echo "No files found in $source_directory"
fi
printf "\n"

Guide for using the Bash script

The “automate.sh” script needs to be run from a Unix environment, or from cygwin if run on Windows. It will look for files in the specified directory, then upload each of them to the specified server. You will need to change the following variables listed at the top of the script:

  • username: your dashboard username for logging onto the web interface
  • password: your dashboard password for logging onto the web interface
  • organisation: your organisation name exactly as in the dashboard URL
  • format: your file format identifier exactly as provided
  • source_directory: your local directory containing data files to be uploaded
  • archive_directory: your local directory for moving data files once uploaded

Automating on Unix

Within a Unix environment you can set the script to run as a cron job, for example add this to /etc/crontab:

0 10 * * * user /usr/local/bin/automate.sh >> /usr/var/log/edi-net.log 2>&1

This will run the script once a day as ‘user’ at 10am and send the output to a log file ‘edi-net.log’.

Automating on Windows:

From Windows/cygwin you can use the Windows Task Scheduler

  1. Open Task Scheduler, right-click on an empty space and select ‘new task’
  2. Go to the ‘Triggers’ tab and select ‘Daily’ and set a time for it to run
  3. Go to the ‘Actions’ tab, and click ‘new’
  4. In the ‘Program to run’ text field put the path to the cygwin shell (bash). Normally: c:\\cygwin\bin\bash.exe or c:\\cygwin64\bin\bash.exe
  5. In the ‘arguments’ field enter -l -c followed by the path to the script and the location of a log file - all in double quotes. NOTE: the argument needs a Unix style path name with forward slashes not back slashes, and the path name relative to cygwin - so the c drive is: /cygdrive/c rather than c:\\. So, if the script and log file are in the root of the C drive: "-l -c /cygdrive/c/automate.sh >> /cygdrive/c/edi-net_log.txt 2>&1"
  6. And finally in the ‘Start in’ field enter the path to /bin in cygwin: c:\\cygwin\bin or c:\\cygwin64\bin.

This will now run the script daily at the specified time and log the output to the specified file.

The Interwatt CSV format is now supported for those who prefer to keep their interwatt API hidden from the internet.

image