REST Web Services using Json and requiring authentication

But first…

Registration for NAV TechDays 2017 have been opened.  I will do a workshop on web services and json.  I will be using both C/AL and AL with VS Code in this workshop.

Make sure to register for the conference and if possible go to one or two of the workshops.

Now to the topic.  Yesterday I started to develop an integration solution for bokun.io.  Their API is RESTful and uses Json file formats.  It also requires authentication.

In a project like this I usually start by using the OCR Service Setup from standard NAV.  Create a Setup table and a page.

Looking at the API documentation we can see that we need to use HmacSHA1 with both Access Key and Secret Key to authenticate.  In other project I used HmacSHA256 with the Access Key for the Azure API.

First part of the authentication is the time stamp created in UTC.  I find it easy to use the DateTime DotNet variable to solve this.  There are two different formatting I needed to use.

REST service normally just use GET or POST http methods.  The authentication is usually in the request headers.  This is an example from bokun.is

The GetSignature function is

The Secret Key string and the Signature is converted to a byte array.  The Crypto class is constructed with the Secret Key Byte Array and used to compute hash for the Signature Byte Array. That hash is also a byte array that must be converted to a base64 string.  This will give you the HmacSHA1 signature to use in the request header.

My Azure project is using HmacSHA256 but the code is similar.

Azure displays the Access Keys in base64 format while bokun.is has a normal string.

A little further down the line I choose not to use XML Ports, like I did here, but still convert Json to Xml or Xml to Json.

I use the functions from Codeunit “XML DOM Management” to handle the Xml.  This code should give you the general idea.

OBJECT Codeunit 60201 Bokun.is Data Management
{
  OBJECT-PROPERTIES
  {
    Date=;
    Time=;
    Version List=;
  }
  PROPERTIES
  {
    OnRun=BEGIN
          END;

  }
  CODE
  {
    VAR
      XMLDOMMgt@60200 : Codeunit 6224;

    PROCEDURE ReadCurrencies@1(ResponseString@10035985 : Text;VAR CurrencyBuffer@10035988 : TEMPORARY Record 60201);
    VAR
      XmlDocument@60202 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlDocument";
      ResultXMLNodeList@60201 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlNodeList";
      ResultXMLNode@60200 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlNode";
    BEGIN
      XmlDocument := XmlDocument.XmlDocument;
      XmlDocument.LoadXml(JsonToXml('{"Currency":' + ResponseString + '}'));
      XMLDOMMgt.FindNodes(XmlDocument.DocumentElement,'Currency',ResultXMLNodeList);
      FOREACH ResultXMLNode IN ResultXMLNodeList DO
        ReadCurrency(ResultXMLNode,CurrencyBuffer);
    END;

    LOCAL PROCEDURE ReadCurrency@60205(ResultXMLNode@60201 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlNode";VAR CurrencyBuffer@60200 : TEMPORARY Record 60201);
    BEGIN
      WITH CurrencyBuffer DO BEGIN
        INIT;
        Code := XMLDOMMgt.FindNodeText(ResultXMLNode,'code');
        "Currency Factor" := ToDecimal(XMLDOMMgt.FindNodeText(ResultXMLNode,'rate'));
        Payment := ToBoolean(XMLDOMMgt.FindNodeText(ResultXMLNode,'payment'));
        INSERT;
      END;
    END;

    PROCEDURE ReadActivities@60201(ResponseString@10035985 : Text);
    VAR
      XmlDocument@60202 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlDocument";
      ResultXMLNodeList@60201 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlNodeList";
      ResultXMLNode@60200 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlNode";
    BEGIN
      XmlDocument := XmlDocument.XmlDocument;
      XmlDocument.LoadXml(JsonToXml(ResponseString));
    END;

    PROCEDURE GetActivityRequestJson@10035986(NoOfParticipants@60200 : Integer;StartDate@60201 : Date;EndDate@60202 : Date) Json : Text;
    VAR
      XmlDocument@10035987 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlDocument";
      CreatedXMLNode@10035988 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlNode";
    BEGIN
      XmlDocument := XmlDocument.XmlDocument;
      XMLDOMMgt.AddRootElement(XmlDocument,GetDocumentElementName,CreatedXMLNode);
      IF NoOfParticipants <> 0 THEN
        XMLDOMMgt.AddNode(CreatedXMLNode,'participants',FORMAT(NoOfParticipants,0,9));
      IF StartDate <> 0D THEN
        XMLDOMMgt.AddNode(CreatedXMLNode,'startDate',FORMAT(StartDate,0,9));
      IF EndDate <> 0D THEN
        XMLDOMMgt.AddNode(CreatedXMLNode,'endDate',FORMAT(EndDate,0,9));
      Json := XmlToJson(XmlDocument.OuterXml);
    END;

    PROCEDURE XmlToJson@94(Xml@10035985 : Text) Json : Text;
    VAR
      JsonConvert@10017292 : DotNet "'Newtonsoft.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'.Newtonsoft.Json.JsonConvert";
      JsonFormatting@10017296 : DotNet "'Newtonsoft.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'.Newtonsoft.Json.Formatting";
      XmlDocument@10017291 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlDocument";
    BEGIN
      XmlDocument := XmlDocument.XmlDocument;
      XmlDocument.LoadXml(Xml);
      Json := JsonConvert.SerializeXmlNode(XmlDocument.DocumentElement,JsonFormatting.Indented,TRUE);
    END;

    PROCEDURE JsonToXml@95(Json@10035985 : Text) Xml : Text;
    VAR
      JsonConvert@10017293 : DotNet "'Newtonsoft.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'.Newtonsoft.Json.JsonConvert";
      XmlDocument@10017291 : DotNet "'System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.System.Xml.XmlDocument";
    BEGIN
      XmlDocument := JsonConvert.DeserializeXmlNode(Json,GetDocumentElementName);
      Xml := XmlDocument.OuterXml;
    END;

    LOCAL PROCEDURE GetDocumentElementName@97() : Text;
    BEGIN
      EXIT('Bokun.is');
    END;

    LOCAL PROCEDURE ToDecimal@98(InnerText@10035985 : Text) Result : Decimal;
    BEGIN
      IF NOT EVALUATE(Result,InnerText,9) THEN EXIT(0);
    END;

    LOCAL PROCEDURE ToInteger@92(InnerText@10035985 : Text) Result : Decimal;
    BEGIN
      IF NOT EVALUATE(Result,InnerText,9) THEN EXIT(0);
    END;

    LOCAL PROCEDURE ToBoolean@91(InnerText@10035985 : Text) Result : Boolean;
    BEGIN
      IF NOT EVALUATE(Result,InnerText,9) THEN EXIT(FALSE);
    END;

    LOCAL PROCEDURE ToDate@93(InnerText@10035985 : Text) Result : Date;
    BEGIN
      IF NOT EVALUATE(Result,COPYSTR(InnerText,1,10),9) THEN EXIT(0D);
    END;

    LOCAL PROCEDURE ToDateTime@99(InnerText@10035985 : Text) Result : DateTime;
    BEGIN
      IF NOT EVALUATE(Result,InnerText,9) THEN EXIT(0DT);
    END;

    BEGIN
    END.
  }
}

 

 

Blob Data and TRANSFERFIELDS

As I was reading the list of published platform updates for NAV 2013 I saw that Microsoft was fixing a bug where the content of a BLOB field is not transferred to a new record with the TRANSFERFIELDS function.  I must admit that I have never counted on the TRANSFERFIELDS function to move the BLOB data from one table to another.  I have always created in- and outstream like this

[code]
InboxTransaction2.CALCFIELDS("PDF Document");
IF "InboxTransaction2.PDF Document".HASVALUE THEN BEGIN
InboxTransaction2."PDF Document".CREATEINSTREAM(InStr);
HandledInboxTransaction2."PDF Document".CREATEOUTSTREAM(OutStr);
COPYSTREAM(OutStr,InStr);
END;[/code]

Now, If I am using a version of NAV 2013 that has fixed the TRANSFERFIELDS bug I can simply do

[code]
InboxTransaction2.CALCFIELDS("PDF Document");
HandledInboxTransaction2."PDF Document" := InboxTransaction2."PDF Document";[/code]

or

[code]
InboxTransaction2.CALCFIELDS("PDF Document");
HandledInboxTransaction2.TRANSFERFIELDS(InboxTransaction2);[/code]

Keep in mind that if the CALCFIELDS function is missing then nothing will be transferred.

UPLOAD and DOWNLOAD size limit

Always something new.  I have bin using the Record Links to store pointers to scanned files, created pdf files all other files that the user would like to link to a record and store centrally.  I have the option to import the file to a BLOB field and also to a separate tables in a different database.

What I do next is to install a single aspx page to an internal web server and the URL in the Record Links table points me to this page with parameters that define where to get the centrally stored file.  This aspx page can fetch the file from the database or through web services from NAV BLOB field.

My first version of this was to have everything executed on the client side.  I did not see that working for NAV 2013 where we can have users that authenticate outside the windows domain.  The same is for NAV 2009 R2 where the Role Tailored Client is connecting to the service tier via WAN.

So I moved the functionality to the service tier and wanted to use UPLOAD and DOWNLOAD to transfer the files to and from the client.  Here I hit a wall.  There is a size limit to the files that I can upload and download.

I therefore looked at the code I had created earlier and modified it a little bit.

[code]PROCEDURE UploadClientFile@1100408005(FileName@1100408000 : Text[1024]) ServerFileName : Text[1024];
VAR
Document@1100408011 : BigText;
ServerConvert@1100408010 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.Convert";
ServerBase64File@1100408009 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.File";
ServerDocumentFile@1100408008 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.File";
ClientConvert@1100408007 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.Convert" RUNONCLIENT;
ClientFile@1100408006 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.File" RUNONCLIENT;
ServerFileStream@1100408005 : File;
ServerBase64FileName@1100408004 : Text[1024];
OutStr@1100408002 : OutStream;
BEGIN
Document.ADDTEXT(ClientConvert.ToBase64String(ClientFile.ReadAllBytes(FileName)));
ServerBase64FileName := ThreeTier.ServerTempFileName(”,”);
ServerFileStream.WRITEMODE(TRUE);
ServerFileStream.CREATE(ServerBase64FileName);
ServerFileStream.CREATEOUTSTREAM(OutStr);
Document.WRITE(OutStr);
ServerFileStream.CLOSE;

ServerFileName := ThreeTier.ServerTempFileName(”,”);

ServerDocumentFile.WriteAllBytes(
ServerFileName,
ServerConvert.FromBase64String(
ServerBase64File.ReadAllText(ServerBase64FileName)));

ServerBase64File.Delete(ServerBase64FileName);
END;

PROCEDURE DownloadServerFile@1100408017(FileName@1100408000 : Text[1024]) ClientFileName : Text[1024];
VAR
Document@1100408002 : BigText;
ServerFile@1100408003 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.File";
ServerConvert@1100408001 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.Convert";
ClientBase64File@1100408006 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.File" RUNONCLIENT;
ClientStreamWriter@1100408007 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.StreamWriter" RUNONCLIENT;
ClientConvert@1100408010 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.Convert" RUNONCLIENT;
ClientFile@1100408009 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.File" RUNONCLIENT;
ClientPath@1100408011 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.Path" RUNONCLIENT;
ClientBase64FileName@1100408008 : Text[1024];
SubString@1100408004 : Text[1024];
Pos@1100408005 : Integer;
BEGIN
Document.ADDTEXT(ServerConvert.ToBase64String(ServerFile.ReadAllBytes(FileName)));
ClientBase64FileName := ThreeTier.ClientTempFileName(”,’b64′);
ClientStreamWriter := ClientBase64File.CreateText(ClientBase64FileName);
Pos := 1;
WHILE Pos < Document.LENGTH DO BEGIN
Pos := Pos + Document.GETSUBTEXT(SubString,Pos,MAXSTRLEN(SubString));
ClientStreamWriter.Write(SubString);
END;
ClientStreamWriter.Close;

ClientFileName :=
ClientPath.GetTempPath +
SigningTools.RemovePath(FileName);

ClientFile.WriteAllBytes(
ClientFileName,
ClientConvert.FromBase64String(
ClientBase64File.ReadAllText(ClientBase64FileName)));
END;
[/code]

The problem is that this method is not fast enough. Perhaps someone reading this has a faster solution.  I convert the file to base64 and use BigText variable to move it from and to the service tier.  This will support large files but it is slow.

Using a .dll proxy for web services

I have now completed my first all-dotnet codeunit.  The codeunit uses a dll file that I created from the web service WDSL.  This makes the programming a lot easier.

This solution has a BLOB fields that stores both incoming and outgoing xml.  When I use a proxy dll I don’t build a xml document and I never handle xml documents.  Again dotnet has a solution.  I use xml serializer from the system.xml object.

[code]LOCAL PROCEDURE SerializeToXMLStream@1100408008(VAR Object@1100408002 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.Object" RUNONCLIENT;VAR NavOutstr@1100408004 : OutStream);
VAR
xmlSerializer@1100408000 : DotNet "’System.Xml, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.Xml.Serialization.XmlSerializer" RUNONCLIENT;
StreamWriter@1100408003 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.StreamWriter" RUNONCLIENT;
File@1100408001 : DotNet "’mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′.System.IO.File" RUNONCLIENT;
NavInstr@1100408005 : InStream;
BEGIN
IF ISNULL(Object) THEN EXIT;
StreamWriter := StreamWriter.StreamWriter(TempFileName);
xmlSerializer := xmlSerializer.XmlSerializer(Object.GetType);
xmlSerializer.Serialize(StreamWriter,Object);
StreamWriter.Close;
UPLOADINTOSTREAM(”,MagicPath,”,TempFileName,NavInstr);
COPYSTREAM(NavOutstr,NavInstr);
File.Delete(TempFileName);
END;[/code]

I can send the proxy object to this function and will get the xml into the BLOB field.

[code]CurrencyRates := StatementService.GetCurrencyRates(TypeRates,CREATEDATETIME(RatesDate,235959T));

WITH BankAccAction DO BEGIN
"Modified by User ID" := USERID;
"Modification Date and Time" := CURRENTDATETIME;
"Incoming Message".CREATEOUTSTREAM(OutStr);
SerializeToXMLStream(CurrencyRates,OutStr);
MODIFY;
COMMIT;
END;[/code]

This example is executed on the client since I am using certificates and web services security for the communication.

UTF-8 to ISO-8859-1

I need to create a XML document with ISO-8859-1 encoding.  This option is not available in XML Ports so what can you do?

I wrote the following function to read an UTF-8 encoded file and convert it to ISO-8859-1.
[code htmlscript=”false”]PROCEDURE UFT8File_To_ISO88591File@1200050006(UFT8_FileName@1200050000 : Text[1024];ISO88591_FileName@1200050001 : Text[1024]);
VAR
UFT8Stream@1200050008 : Automation "{2A75196C-D9EB-4129-B803-931327F72D5C} 2.8:{00000566-0000-0010-8000-00AA006D2EA4}:’Microsoft ActiveX Data Objects 2.8 Library’.Stream";
ISOStream@1200050007 : Automation "{2A75196C-D9EB-4129-B803-931327F72D5C} 2.8:{00000566-0000-0010-8000-00AA006D2EA4}:’Microsoft ActiveX Data Objects 2.8 Library’.Stream";
String@1200050009 : Text[1024];
adReadAll@1200050002 : Integer;
adSaveCreateOverWrite@1200050003 : Integer;
adTypeBinary@1200050004 : Integer;
adTypeText@1200050005 : Integer;
adWriteChar@1200050006 : Integer;
BEGIN
adReadAll := -1;
adSaveCreateOverWrite := 2;
adTypeBinary := 1;
adTypeText := 2;
adWriteChar := 0;

CREATE(UFT8Stream);
CREATE(ISOStream);
UFT8Stream.Open;
UFT8Stream.Type := adTypeBinary;
UFT8Stream.LoadFromFile(UFT8_FileName);
UFT8Stream.Type := adTypeText;
UFT8Stream.Charset := ‘UTF-8’;

ISOStream.Open;
ISOStream.Type := adTypeText;
ISOStream.Charset := ‘iso-8859-1′;
WHILE NOT UFT8Stream.EOS DO BEGIN
String := UFT8Stream.ReadText(MAXSTRLEN(String));
ISOStream.WriteText(String,adWriteChar);
END;
ISOStream.SaveToFile(ISO88591_FileName,adSaveCreateOverWrite);
ISOStream.Close;
CLEAR(ISOStream);
UFT8Stream.Close;
CLEAR(UFT8Stream);
END;[/code]
Then I stream the BLOB that holds the UTF-8 encoded XML to a file and convert it.
[code htmlscript=”false”]IF ISCLEAR(SystemShell) THEN
CREATE(SystemShell);

Log."Outgoing Message".CREATEINSTREAM(InStr);
UFT8_FileName := ThreeTierMgt.ServerTempFileName(”,’XML’);
PaySlipFile.CREATE(UFT8_FileName);
PaySlipFile.CREATEOUTSTREAM(OutStr);
COPYSTREAM(OutStr,InStr);
PaySlipFile.CLOSE;
IF ISSERVICETIER THEN BEGIN
ISO_FileName := ThreeTierMgt.ServerTempFileName(”,’XML’);
Helper.UFT8File_To_ISO88591File(UFT8_FileName,ISO_FileName);
PaySlipFile.OPEN(ISO_FileName);
PaySlipFile.CREATEINSTREAM(InStr);
DOWNLOADFROMSTREAM(InStr,Text006,”,Text007,ToFile);
SystemShell.DeleteFile(ISO_FileName);
END ELSE BEGIN
ISO_FileName := ToFile;
Helper.UFT8File_To_ISO88591File(UFT8_FileName,ISO_FileName);
END;
SystemShell.DeleteFile(UFT8_FileName);[/code]
Manually you will need to replace the first line in the XML Document from ‘<?xml version=”1.0″ encoding=”UTF-8″ ?>’ to ‘<?xml version=”1.0″ encoding=”ISO-8859-1″?>’

 

Loading a client file into BLOB with RTC Client

For some time have been looking for a solution on how to upload a file into BLOB with RTC Client.  The built in functions, UPLOAD and UPLOADINTOSTREAM both force an Open Dialog unless you first copy the file to a temporary path.  I already had the file name and wanted to skip that part.

I always stopped on the fact that I was unable to move binary data with code from the client layer to the server layer.  Then today, I finally got an idea on how to solve this.  I convert the client file to a base64 string, transfer that string to the server layer and save as a file.  Then I create a binary server file based on the base64 server file.  That file is identical to the client file and ready to be imported into BLOB on the server side.
[code htmlscript=”false”]IF ISSERVICETIER THEN BEGIN
Document.ADDTEXT(
ClientConvert.ToBase64String(
ClientFile.ReadAllBytes(ImageFileName)));
ServerBase64FileName := ThreeTireMgt.ServerTempFileName(”,”);
ServerFileStream.WRITEMODE(TRUE);
ServerFileStream.CREATE(ServerBase64FileName);
ServerFileStream.CREATEOUTSTREAM(OutStr);
Document.WRITE(OutStr);
ServerFileStream.CLOSE;

ServerDocumentFileName := ThreeTireMgt.ServerTempFileName(”,”);
ServerDocumentFile.WriteAllBytes(
ServerDocumentFileName,
ServerConvert.FromBase64String(
ServerBase64File.ReadAllText(ServerBase64FileName)));

ServerFileStream.OPEN(ServerDocumentFileName);
ServerFileStream.CREATEINSTREAM(InStr);
Image.CREATEOUTSTREAM(OutStr);
COPYSTREAM(OutStr,InStr);
ServerFileStream.CLOSE;
ServerBase64File.Delete(ServerBase64FileName);
ServerDocumentFile.Delete(ServerDocumentFileName);
END ELSE BEGIN
Image.IMPORT(ImageFileName,FALSE);
END;[/code]
This should support files upto 1.5GB in size.

ImageTest