Tuesday, November 19, 2013

Base64 encode/decode part of messages in map

This post is to show how to base64 encode or decode a part of a message (or the entire input message for that matter) and output the resulting string in an element in the output message.

The technique used allows for execution of inline C# (or VB) code from another script functoid using inline XSLT / XSLT Template.

The input and output schemas looks like this:




In the example, our entire input message should be base64 encoded and the resulting string output to the EncodedData element in the output message. This element is of the type xs:string.

First, we create a script functoid to hold our inline C# code for base64 encoding of the input node. When building such a method, one might first try to use string as the datatype for the input which will result in just the inner xml being transferred to the method. This is since the data from our XSLT call will be of the type MS.Internal.Xml.Cache.XPathDocumentNavigator. We therefore declare the input variable as XPathNavigator instead to get access to the full XML node. The base64 encoding is done using the System.Text.Encoding class together with the System.Convert class.

public string Base64EncodeData(XPathNavigator param1)
{
    byte[] bytesToEncode =

        System.Text.Encoding.UTF8.GetBytes(param1.OuterXml);
    string encodedText = Convert.ToBase64String(bytesToEncode);

    return encodedText;
}


This functoid will not be connected to any node, but just left to "hang" in the map since we call it manually from another script functoid. Another solution to have it a bit cleaner would be to place this code in an external assembly and call that, but sometimes it is nice to have everything in the map.

Next we create a script functoid and connect it to the destination element. We set the following Inline XSLT code:

<xsl:variable name="data">
  <xsl:copy-of select="/" />
</xsl:variable>

<xsl:element name="EncodedData">
  <xsl:value-of select="userCSharp:Base64EncodeData($data)" />
</xsl:element>

We first declare a variable that holds a copy of our source node which we simply specify using xpath.
Then we call the previously configured method doing the encoding, and send the resulting data to our created destination element.

The map will now look like this:



The result from sending in a test XML is an XML with the entire original contents base64 encoded in an element.



Wednesday, November 6, 2013

Pay attention when running Biztalk Server and SQL Server on servers in different timezones

This post discusses what most likely is a seldom encountered setup, but nevertheless a possible one. Consider a global company that is using BizTalk. For various reasons, the SQL cluster will have Windows configured with another timezone than the rest of the servers.


This is our mock setup. BizTalk is located in the U.S. and at UTC-5 while the SQL server is in Sweden at UTC+1.

During normal operations, this will work just fine. Each server will locally run in it's own timezone and all is good. The issues appear when working with service windows.

Let's create a receive port with a corresponding receive location pulling in files from a folder.

Now we want to use a service window on this receive location to only allow files to be processed during two hours every afternoon. This is in a real world scenario most likely going to be set up with a time reference to the BizTalk Server, i.e. the service window time will match the timezone on the BizTalk Server. In our example, it's 16:00 to 18:00 that is allowed, UTC+1.



If we look in the corresponding table in the SQL Server for this receive location, we can see that the time is set accordingly.



However, when the BizTalk Server rolls up to this time, the receive location will still not pull the file in! Why is that?

Well, it seems that when BizTalk is checking if the receive location should be active or not, it compares the stored service window time with the local time on the SQL Server. BizTalk is after all heavily based on SQL Server. This have the effect that the time that we set on the BizTalk machine in UTC+1 and which is stored as such but without the timezone information will be used as is and hence be offset by six hours in this example.

If we change the service window time in the BizTalk Administration Console to be offset with the corresponding six hours (i.e. 16:00 will be 10:00 instead), the receive location will trigger at the intended time.

So far so good. If you know about this issue, it is easy to work around it. However, if you add in the normal complexity regarding daylight saving time that will change the offset yet a little twice a year, you are most likely heading towards a small headache!

Wednesday, October 30, 2013

Poor-man's scheduled trigger message task in BizTalk

From time to time you might encounter the need for a trigger message in BizTalk. No matter the reason, you need a message that is sent to BizTalk every x minutes that in turn will trigger a process.

In most cases, you will end up installing the BizTalk Scheduled Task Adapter from Codeplex. This adapter is throughly tested and well-used in different scenarious around the world and is a very valid option.

If you do not want to install a third-party adapter for any reason but still need a trigger message to be sent every x minutes, you can script it using the Windows Task Scheduler. This solution will however require you to add the scheduled task to your monitoring solution to make sure that it stays active.

Yet another solution that can be commonly seen is to have an SQL Agent job set a flag in a table every x minutes and then have BizTalk poll this table according to the usual techniques.

My solution to creating a simple trigger message is to utilize the built-in features in BizTalk and not depend on any external trigger mechanism. By setting up a simple receive location with the WCF-SQL adapter, it can be made.

First, create a receive port and location. In the WCF-SQL transport properties, set the endpoint URI to any valid SQL server. I use BizTalks management database in this case, since there won't be any load on it at all from this receive location and it always will be available.


Next, in the Binding section, set the XmlStoreProcedureRootNodeName to TriggerRoot and the XmlStoredProcedureRootNodeNamespace to http://trigger/v1

Using XmlPolling as InboundOperationType, you set the PolledDataAvailableStatement to Select 1 to simply always trigger a polling statement execution. The PollingStatement in turn is set to WITH XMLNAMESPACES(DEFAULT 'http://trigger/v1') select CURRENT_TIMESTAMP for xml raw ('Trigger'), ELEMENTS

PollingIntervalInSeconds can be set to any interval you like. Also make sure that the PollWhileDataFound property is set to False.








What this does is that when the receive location is started, the first statement will return 1, telling BizTalk that there is a message to be fetched and to execute the polling statement. This statement will return a raw xml message that will look like this:

<TriggerRoot xmlns="http://trigger/v1">
    <Trigger>2013-09-14T12:20:11.037</Trigger>
</TriggerRoot>

You can then create a schema for this message and then in turn use it in your BizTalk solution to trigger a process on a timely schedule. Quick and simple and does the job. The bonus is that as long as the receive location is up and running, you know that it will create a message for you.

Wednesday, October 16, 2013

Captain Obvious to the rescue!


The serverRuntime@appConcurrentRequestLimit setting is being exceeded.

Most likely causes: Setting is too low.
Things you can try: Try increasing the value of the setting.

Well, duh.

Tuesday, August 20, 2013

Connecting BizTalk to Oracle RIB using JMS a.k.a netsh/netmon to the rescue

I was tasked with setting up a connection between BizTalk and an Oracle RIB installation using JMS. The JMS connection was handled using the JNBridge adapter.

Hooking up BizTalk to JMS doesn't pose too much of an issue thanks to good documentation from JNBridge. Most of the issues that occur is about getting proper access to the servers, having the correct parameters for the JMS topics, setting the Java parameters right and so on.

As soon as those things got properly aligned, we successfully received messages published in the RIB to BizTalk. The next part was to send messages to the RIB.

We could see messages leave BizTalk successfully, but nothing ever appeared in the JMS topic. No errors were logged anywhere on either side.

After some error searching, I turned to do a trace using netsh.exe and analyzing this in NetMon. This showed that we had some handshaking going on coupled with the adapter running checks against the JMS setup to make sure that we could put a message on the topic. This resulted in an ORA-24033: no recipients for message error.

Unfortunately, this error message is eaten up by the framework and not forwarded to the adapter so that it can be logged properly. The only way to see it is by running a trace on the traffic.

The error message we got is explained in the documentation as:
ORA-24033: no recipients for message
Cause: An enqueue was performed on a queue that has been set up for multiple dequeuers but there were neither explicit recipients specified in the call nor were any queue subscribers determined to be recipients for this message.
Action: Either pass a list of recipients in the enqueue call or add subscribers to the queue for receiving this message.

I looked around for a solution to this error since we couldn't find any issues with the topic itself. There were a subscriber properly set up and no issues regarding access were to be found.

In an older blog post I could find something to point me in the right direction. In order to send messages to Oracle RIB, you have to set a threadValue parameter in the JMS header (which can be read about in the RIB documenation if you know where to look). This is in turn used in the subscription rule on the other end. If this parameter isn't set, there is no subscriber available since it is a mandatory field in the selector.

After setting a threadValue element to "1" using a custom pipeline component, the messages was successfully sent to the RIB which both can be seen in the topic itself as in the trace where the update to the table is visible with the message as a parameter.


NetMon or a similar tool is invaluable when tracking down similar issues. It is not the first time that NetMon (or WireShark) has provided information that is needed in order to get things to work.

Tuesday, June 18, 2013

Leaving password blank in BizTalk 2013 SFTP adapter will cause host instance to run at 100% CPU without warning

A very interesting tidbit appeared on a server running BizTalk 2013 and using the brand new SFTP adapter that is bundled with the product.

As soon as a file was sent to an SFTP server, the host instance running the adapter would spin up to 100% CPU and stay there. The file would not be sent and no warnings or errors written to the event log.

After being tasked with trying to find out the issue behind this, I started with the normal tasks coupled with a host spinning up without any reason: checking the tracking settings, looking at the messagebox regarding rouge messages, zombies etc. Nothing could be found.

It wasn't until I tried to set up a copy of the flow on another 2013 server that I noticed that the password field in the Send Port properties was blank. Even with an entered password of "" the password field should display a bunch of stars to mask the password (or lack of one).



I set the password to the correct one, and started the host again. Now the file was sent and the host stayed at normal CPU load.

When looking into this, I noticed that when exporting the binding for the send port when the password wasn't set, the CustomProps in the TransportTypeData element never included the Password element. When setting either a blank password or a proper one, the binding will include a password placeholder of which can be translated to NULL, i.e. there is no password defined (it is never written to the binding file for security reasons). But if you never set a password, this field is never written to the binding file at all!

This is important to be aware of since I also noticed that if you try to set the password to "blank", by entering a character and then deleting it again and then saving, it is not entered since the GUI will interpret it as nothing has changed and therefore not set the password field to blank, leaving it completely undefined. Instead you have to enter a dummy character, save, enter the property dialog again and delete the character before doing the final save. That way the blank password will be properly saved.

Wednesday, May 8, 2013

Remember to set the child delimeter even for single element lines in a flat file schema if using wrap characters

I briefly ran into this when creating a larger flat file schema for a message. The input message had line tags that I could use to identify the different rows. Most of the rows had several distinct fields to read, but there were rows with only the tag and a string of text encapsulated in quotation marks.

As an example, we'll use an input file like so:

#FIRST 12345 "This is the first string"
#SECOND "This is the second string"


This can then be used to create a schema which identifies the two distincs rows based on their tag (#FIRST and #SECOND respectively) and then split the fields on a delimiter of 0x20 (the space character).

The schema can be like this:

 <?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://TestProject.Schemas" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" elementFormDefault="qualified" targetNamespace="http://TestProject.Schemas" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:annotation>
    <xs:appinfo>
      <schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
      <b:schemaInfo standard="Flat File" codepage="65001" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" root_reference="Root" />
    </xs:appinfo>
  </xs:annotation>
  <xs:element name="Root">
    <xs:annotation>
      <xs:appinfo>
        <b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xD 0xA" child_order="postfix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
      </xs:appinfo>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:annotation>
          <xs:appinfo>
            <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
          </xs:appinfo>
        </xs:annotation>
        <xs:element name="First">
          <xs:annotation>
            <xs:appinfo>
              <b:recordInfo tag_name="#FIRST" structure="delimited" child_delimiter_type="hex" child_delimiter="0x20" child_order="prefix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
            </xs:appinfo>
          </xs:annotation>
          <xs:complexType>
            <xs:sequence>
              <xs:annotation>
                <xs:appinfo>
                  <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
                </xs:appinfo>
              </xs:annotation>
              <xs:element name="Id" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo justification="left" sequence_number="1" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
              <xs:element name="Text" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo justification="left" sequence_number="2" wrap_char_type="char" wrap_char="&quot;" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="Second">
          <xs:annotation>
            <xs:appinfo>
              <b:recordInfo tag_name="#SECOND" structure="delimited" child_order="prefix" sequence_number="2" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
            </xs:appinfo>
          </xs:annotation>
          <xs:complexType>
            <xs:sequence>
              <xs:annotation>
                <xs:appinfo>
                  <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
                </xs:appinfo>
              </xs:annotation>
              <xs:element name="Text" type="xs:string">
                <xs:annotation>
                  <xs:appinfo>
                    <b:fieldInfo justification="left" sequence_number="1" wrap_char_type="char" wrap_char="&quot;" />
                  </xs:appinfo>
                </xs:annotation>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>


This will however generate an output like so:

<Root xmlns="http://TestProject.Schemas">
    <First>
        <Id>12345</Id>
        <Text>This is the first string</Text>
    </First>
    <Second>
        <Text>"This is the second string"</Text>
    </Second>
</Root>


Notice the quotation marks that are still left in the string even if we have defined them as a wrap character in the schema for that field.

In order to make them disappear from the field properly in the same manner as in the first line of text, we have to set the Child Delimiter parameter on the "Second" child record:

 <xs:element name="Second">
    <xs:annotation>
        <xs:appinfo>
            <b:recordInfo tag_name="#SECOND" structure="delimited" child_order="prefix" sequence_number="2" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" child_delimiter_type="hex" child_delimiter="0x20" />
        </xs:appinfo>
    ...


Which in turn will make the quotation marks disappear from our element data.

Saturday, March 23, 2013

MSDTC error after cloning a virtual machine

A common development task is to clone virtual servers for different development scenarios. I myself have a bunch of virtual servers configured that I clone in order to get new fresh installs to lab with. However, when cloning machines in a BizTalk setup, you have to be aware of the issues that can develop due to the need for unique identification keys in MSDTC. When cloning a server to two instances and using one for SQL Server and the other as the BizTalk application server, MSDTC will not work. When running DTCTester the following error will appear:


Error: SQLSTATE=24000,Native error=0,msg=[Microsoft][ODBC SQL Server Driver]Invalid cursor state
 The reason in this case is that the CID GUID is not unique on the servers.

Run regedit and browse to HKEY_CLASSES_ROOT\CID\ and find the GUID where Description is set to MSDTC (it is one of the four). The GUID should be exactly the same on the other server.

  

In order to fix this, you have to uninstall and then install MSDTC again (or you could go the wild path and just set another GUID). Uninstalling/installing MSDTC is done via the command prompt.

Open a command prompt as administrator and run the command
msdtc -uninstall
followed by
msdtc -install
Verify that the DTC coordinator has started up and check the CID values in the registry. When running MSDTCTester again it should show a successful result:


This error will be caught by the BizTalk MessageBox Viewer. It will show an error of
is duplicated on servers and - Check if < server1> and are not cluster nodes as CID must be unique on non-clustered servers!

Tuesday, February 26, 2013

Script BizTalk Application export to .msi with options using PowerShell

I've had the need to export BizTalk Applications using scripts, but with the requirement to not export web directories or global parties to the .msi file.

This can be done using the GUI:


However, it cannot be done (that I know of) using the btstask command with the exportapp parameter.

I opted for creating a PowerShell script that more or less extends the btstask exportapp command. The script takes a few parameters making it a bit more versatile:

$appName is the name of the BizTalk application to export
$exportPath is the path to write the .msi file to
$exportWebDir is a boolean indicating whether to also export web directories or not
$removeDefaultBinding is a boolean controlling whether the default bindings will be written to the .msi file or not. This is handy to remove in those cases you want to just keep bindings for dev/test/production environments.
$exportGlobalParties is a boolean controlling whether to include the Global Parties in the binding file or not. These will of course not be written if for instance you only have the default bindings and set the parameter $removeDefaultBinding to false and $exportGlobalParties to true, since the binding file never will be written at all.

The script basically dumps a ResourceSpecification file which then gets modified depending on the parameters entered. This ResourceSpecification is then used as input to the btstask exportapp command creating an .msi file according to our needs.

The full script looks like this:

param($appName, $exportPath, $exportWebDir, $removeDefaultBinding, $exportGlobalParties)

Write-Output "Exporting ResourceSpec..."
BTSTask ListApp /ApplicationName:$appName /ResourceSpec:$pwd\ResourceSpecTemp.xml

If (!($?))
{
    throw "Could not export resource specification. Verify application name."
}

Write-Output "Reading ResourceSpec..."
$xmlResource = [xml] (Get-Content $pwd\ResourceSpecTemp.xml)

If ($exportWebDir -eq $false)
{
    Write-Output "Removing web directories..."
    $delnodes = $xmlResource.SelectNodes("/*[local-name()='ResourceSpec' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resources' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resource' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12'][@Type='System.BizTalk:WebDirectory']")

    ForEach($delnode in $delnodes)
    {
        [void]$xmlResource.ResourceSpec.Resources.RemoveChild($delnode)
    }
}

If ($removeDefaultBinding -eq $true)
{
    Write-Output "Removing Default binding info..."
    $delnodes = $xmlResource.SelectNodes("/*[local-name()='ResourceSpec' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resources' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12']/*[local-name()='Resource' and namespace-uri()='http://schemas.microsoft.com/BizTalk/ApplicationDeployment/ResourceSpec/2004/12'][@Type='System.BizTalk:BizTalkBinding'][@Luid='Application/$appName']")

    ForEach($delnode in $delnodes)
    {
        [void]$xmlResource.ResourceSpec.Resources.RemoveChild($delnode)
    }
}

Write-Output "Saving modified ResourceSpec..."
$xmlResource.Save("$pwd\ResourceSpecTemp.xml")

Write-Output "Exporting application $appName..."

If ($exportGlobalParties -eq $true)
{
    $globalPartiesParam = "/G"
}
else
{
    $globalPartiesParam = ""
}

BTSTask ExportApp /ApplicationName:$appName /Package:$exportPath\$appName.msi /ResourceSpec:$pwd\ResourceSpecTemp.xml $globalPartiesParam

If (!($?))
{
    throw "Could not export application. Verify application name and parameters."
}
       
Write-Output "Cleaning up..."
Remove-Item $pwd\ResourceSpecTemp.xml

Write-Output "DONE!"

Exit 0


And will be called like so:

ExportBtsMsi.ps1 BtsDemo1 "C:\" false false false

Which will result in output like this:



Friday, January 18, 2013

Run Microsoft pre-configured VHD evaluations without Hyper-V using VirtualBox

I needed to run the BizTalk 2010 VHD for demo purposes, but lacked access to a Windows 2008 Server with Hyper-V in order to run the virtual machine.

My first attempt was to simply mount the VHD file in a newly configured Virtual PC machine. This failed with the virtual machine rebooting shortly after start.

Next I tried to create a new virtual machine in VirtualBox, giving me pretty much the same result, but with a bluescreen flashing past just before reboot.




In the bluescreen that quickly flashes by, it is possible to see the error code 0x0000007B, also known as INACCESSIBLE_BOOT_DEVICE. This is an indication of a possible issue with the boot sector, device driver and similar things. It led me to take a peek at the storage settings for the virtual machine that showed me that the IDE controller had no disks attached (but a CD/DVD drive) and my added VHD file was attached to a SATA controller.


I removed the VHD file from the SATA controller and removed the controller completely. I then added a new harddisk to the IDE controller and assigned the VHD file to it, thinking that this most likely was how the VHD file was set up initially.


And indeed it was so. This made the virtual machine start up ok!


I did run into issues when logging on though since the admin password includes the character "@" which I was unable to type due to some issue with VirtualBox and the Alt Gr key on my keyboard that was needed to type the character in. This was however quickly remedied by using the ALT-sequence instead, typing ALT+064 for the "@" character.

After a day of testing, I cannot see any issues in the VM, but everything is running just fine! For those that are using VMWare instead, the same solution should work just as well.