Announcing new version of DNV GL Rules Chatbot

, , , , ,

Happy new year!

I am happy to announce the new version of Rules Chatbot is ready now. The key new features are below:

  • Support full text search (based on SharePoint Search)
  • Indexing DNV GL classification rules documents with enhanced metadata
  • Customized ranking
  • Upgraded to Bot Framework 3.0

URL:
https://dnvgl-rules-bot.azurewebsites.net

live preview

Demo

Architect Overview

The architect of the chat bot is shown in below

1. Bot framework and channels

Bot framework out-of-the-box supports several conversation channels, but you might noticed that they are more consumer based channels. Skype for Business, for example, is not supported at this moment.

2. Chatbot Web API application in Azure

Chatbot Web API is a simple standard ASP.Net Web API project. It has two key class:

  • MessagesController
  • RulesAndStandardsDialog

MessagesController class

This is the end-point for listening all incoming messages, and forward to RulesAndStandardsDialog class

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[BotAuthentication]
public class MessagesController : ApiController
{
    /// <summary>
    /// POST: api/Messages
    /// Receive a message from a user and reply to it
    /// </summary>
    /// 

    public async Task<HttpResponseMessage> Post([FromBody]Activity activity)
    {
        ConnectorClient connector = new ConnectorClient(new Uri(activity.ServiceUrl));
        if (activity != null && activity.Type == ActivityTypes.Message)
        {
            await Conversation.SendAsync(activity, () => new DNVGLBot.Controllers.RulesAndStandardsDialog());
        }
        else
        {
            return null; //HandleSystemMessage
        }
        return new HttpResponseMessage(System.Net.HttpStatusCode.Accepted);
    }

}

RulesAndStandardsDialog class

This class is responsible for consuming the result from LUIS web service and perform actual task. For example, [LuisIntent(“bot.intent.search”)] attribute means that the SearchDocument function will be called once the bot.intent.search intent was identified by LUIS.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[LuisModel("YOUR-LUIS-MODEL-ID", "YOUR-SUBSCRIPTION-ID")]
[Serializable]
public class RulesAndStandardsDialog: LuisDialog<object>
{
	//Some lines of code were omitted 

    [LuisIntent("bot.intent.search")]
    public async Task SearchDocument(IDialogContext context, LuisResult result)
    {
        string keywords = TryFindKeywords(result);

        if (!string.IsNullOrEmpty(keywords))
        {
            await DoSearch(context);
        }
        else
        {
            await context.PostAsync("Sorry, I can not find any keywords, please try again.");
            context.Wait(MessageReceived);
        }
   }

    [LuisIntent("bot.help")]
    public async Task Help(IDialogContext context, LuisResult result)
    {
        await context.PostAsync(GetHelpMessage());
        context.Wait(MessageReceived);
    }

    [LuisIntent("bot.about")]
    public async Task About(IDialogContext context, LuisResult result)
    {
        await context.PostAsync(GetAboutMessage());
        context.Wait(MessageReceived);
    }
}

3. Storage account in Azure

Chatbot is using Azure storage account (table) for storing end user profiles, such as Full name, email address and subscribed rules codes. It is not a mandatory component for the bot, but simply an user profile database.

4. Luis in Cognitive services

Luis model is the brain of the chat bot. It is pretty easy to setup and training it without programming skill. Based on the experience, 100 sample inputs can produce a reasonable AI.

Tips: Remember to re-publish Luis endpoint once a while during the training process, otherwise the client (bot) won’t pick up the newly trained sample.

5. Search engine in SharePoint

Once Luis has identified the search intent and the to-be-search keywords, the information will be passed to the search engine for query. In this case, the search engine is SharePoint.

All rules pdf files are indexed by SharePoint search engine - it provides good support for full text search and meta-data enhancement (via Content Enrichment Web Service during the indexing process).

Using a search engine is the biggest improvement compares to older version, where the search logic was simply matching words in rules title.

I am using SharePoint PnP framework to handle the authentication to the on-premise SharePoint farm, which is protected by ADFS. Also, I have learned from our expert Mikael Svenson to add the freshness boost to the result.

Unfortunately built-in date variables do not work in C#/KeywordQuery scenario, therefore I have rewrite the code as below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
string samlSite = "https://SHAREPOINTURL";

//XRank for boosting fresh content
string daysAgo580 = DateTime.Now.AddDays(-580).ToString("yyyy-MM-dd");
string daysAgo365 = DateTime.Now.AddDays(-365).ToString("yyyy-MM-dd");
string daysAgo180 = DateTime.Now.AddDays(-180).ToString("yyyy-MM-dd");
string daysAgo90 = DateTime.Now.AddDays(-90).ToString("yyyy-MM-dd");
string daysAgo60 = DateTime.Now.AddDays(-60).ToString("yyyy-MM-dd");
string daysAgo30 = DateTime.Now.AddDays(-30).ToString("yyyy-MM-dd");
string daysAgo16 = DateTime.Now.AddDays(-16).ToString("yyyy-MM-dd");
string daysAgo7 = DateTime.Now.AddDays(-7).ToString("yyyy-MM-dd");
string daysAgo1 = DateTime.Now.AddDays(-1).ToString("yyyy-MM-dd");
string daysAgo0 = DateTime.Now.ToString("yyyy-MM-dd");

OfficeDevPnP.Core.AuthenticationManager am = new OfficeDevPnP.Core.AuthenticationManager();
using (ClientContext ctx = am.GetADFSUserNameMixedAuthenticatedContext(samlSite, "USERNAME", "PASSWORD", "DOMAIN", "STS", "IDPID"))
{
    KeywordQuery keywordQuery = new KeywordQuery(ctx);
    keywordQuery.QueryText = string.Format("((((((((({0} contentsource:\"dnvglrules\" XRANK(cb=1) title:\"{0}\")  XRANK(cb=04922713399625873E-17) write>{1}) XRANK(cb=026792479063910794E-18) write>{2}) XRANK(cb=06696008382287308E-17) write>{3}) XRANK(cb=10720794384750529E-17) write>{4}) XRANK(cb=08336806307198713E-17) write>{5}) XRANK(cb=16669442125999617E-17) write>{6}) XRANK(cb=3107141114146401E-16) write>{7}) XRANK(cb=15680891749553738E-17) write>{8}) XRANK(cb=03222684602729131E-17) write>{9}", keyword, daysAgo580, daysAgo365, daysAgo180, daysAgo90, daysAgo60, daysAgo30, daysAgo7, daysAgo1, daysAgo0);

    keywordQuery.RowLimit = 5; //return 5 results everytime, since the dialog screen will be small, e.g. from a mobile
    keywordQuery.StartRow = startRow;
    SearchExecutor searchExecutor = new SearchExecutor(ctx);

    ClientResult<ResultTableCollection> results = searchExecutor.ExecuteQuery(keywordQuery);
    ctx.ExecuteQuery();

    return results;
    
}

Side notes for developer on Windows 10

  1. Enabling Identity Foundation Framework on windows 10
    One of the dependency component of the PnP nuget package is Identity Foundation Framework. You cannot download an installer for windows 10, but simply enable the feature in windows 10.
    Go to Control Panel -> Uninstall a Program on the menu -> Turn Windows Features On or Off -> Check “Windows Identity Framework 3.5”

  2. Install Microsoft.IdentityModel.Extensions.dll library
    Another dependency is Microsoft.IdentityModel.Extensions.dll. Download 32-bit here and 64-bit here.

Final notes

Security is important aspect for any bot service, therefore pay extra attention about how you expose data via your bot service. (Read more at here)

It also the same for the privacy of your end users.

Disclaimer

This chatbot prototype is a personal project. It is only based on the public information of the DNV GL rules. It has NO connection with DNV GL AS. All information from the chatbot is presented ‘as-is’. No warranties whatsoever for correctness; completeness or usefulness.

Share Comments