Planes, trains & automobiles

Having some fun with Custom vision and Blazor
If you just want to check out the code it is here or try the sample here

Introduction

A few days ago I started to build a Magic Mirror (which I will write about when/if it is done :)) and in this process I wanted to use Microsofts Custom Vision Api to identify who is standing in front of it.
When I got it working I decided to see how easy it would be to use this to identify things using its api and Blazor.
In the mirror I have put the model itself onto a Raspberry Pi running Windows 10 IoT, but since Blazor is running inside of Web Assembly in the browser I will use the api provided by Custom vision.

Training a model

The first thing to do is to train a model to recognize images. This can be done on the Custom vision website.

  • Log in to the site using a Microsoft account

  • Create a new project

  • Select a name and select Classification and General (compact)
  • The next step is to add some images to train with, here I selected some planes and tagged them with Airplane
  • Click on the Train-button to start training the model.
  • Once the model is trained we can use it to identify images by clicking Quick test and then selecting View endpoint. This will show the Url and Prediction-key needed to call the api.

The Blazor part

To get started with Blazor, you can follow the Get started guide on the official Blazor site here
Once you have the site up and running, remove all of the pages except Index.cshtml and modify it as follows

@page "/"
@using System.Net.Http.Headers;
@inject HttpClient  _httpClient
<div class="row">  
    <div class="col text-center">
        <h3>BlazorVision - having some fun with <a href="https://customvision.ai">Custom vision</a> and <a href="https://blazor.net">Blazor</a></h3>
        <h5>Check out the code on <a href="https://github.com/bjorndaniel/blazorvision">Github</a></h5>
    </div>
</div>  
<br />  
<div class="row justify-content-center">  
    <div class="col text-center">
        <p>Upload an image to get a prediction</p>
        <input id="fileInput" type="file" onchange="@(() => { State = ImageState.Loading; })" />
    </div>
</div>  
<br />  
<div class="row justify-content-center">  
    <div class="col text-center">
        <img id="imgPreview" style="max-width:400px;" onload="@FileAdded" />
        <input id="hfImage" type="hidden" bind="@Image" />
    </div>
</div>  
<div class="row justify-content-center">  
    <div class="col text-center">
        @switch (State)
        {
            case ImageState.Loading:
                <h3>Loading image...</h3>
                break;
            case ImageState.Analyzing:
                <h3>Analyzing image...</h3>
                break;
            case ImageState.Done:
                <h3>@PredictionResult?.Result</h3>
                break;
            case ImageState.Failure:
                <h3>I'm not sure what this is but probably not a plane, train or automobile...</h3>
                break;
            default:
                <h3></h3>
                break;
        }

    </div>

</div>  
@functions
{
    string Image { get; set; }
    Prediction PredictionResult { get; set; }
    ImageState State { get; set; }
    protected override void OnAfterRender()
    {
        base.OnAfterRender();
        JSInterop.OnLoaded();
    }

    protected async void FileAdded()
    {
        State = ImageState.Analyzing;
        PredictionResult = null;
        StateHasChanged();
        var imageBytes = Convert.FromBase64String(Image.Split(new char[] { ',' })[1]);
        using (var content = new ByteArrayContent(imageBytes))
        {
            _httpClient.DefaultRequestHeaders.Add("Prediction-Key", Configuration.PredictionKey);
            content.Headers.ContentType = new MediaTypeHeaderValue(Configuration.ContentType);
            var response = await _httpClient.PostAsync(Configuration.CustomVisionUrl, content);
            var result = await response.Content.ReadAsStringAsync();
            var predictions = JsonUtil.Deserialize<Predictions>(result);
            var mostLikely = predictions?.predictions.OrderByDescending(_ => _.probability).FirstOrDefault();
            Console.WriteLine(JsonUtil.Serialize(mostLikely));
            if (mostLikely != null && mostLikely.probability > 0.75)
            {
                PredictionResult = mostLikely;
                State = ImageState.Done;
            }
            else
            {
                State = ImageState.Failure;
            }
            StateHasChanged();
        }
    }
}

The page is just a file upload, an img for preview, a hidden field to bind the image-data to and an h3 to show the result.
To make the prediction we need to add a bit of JavaScript to get the image from the file upload to the img-tag. Don't forget to add a script tag pointing to the js-file at the bottom of the index.html!

Blazor.registerFunction('BlazorVision.JSInterop.OnLoaded', () => {  
    document.getElementById('fileInput').addEventListener('change', (e) => {
        var img = document.getElementById('imgPreview'),
            tgt = e.target || window.event.srcElement,
            files = tgt.files,
            fr = new FileReader(),
            hf = document.getElementById('hfImage');
        fr.onload = () => {
            hf.value = fr.result;
            hf.dispatchEvent(new Event('change'));
            img.src = fr.result;
        }
        fr.readAsDataURL(files[0]);
    });
});

The interesting part here is the Blazor.registerFunction part, this allows the javascript to be called from Blazor using interop. This is static methods you can define in C# which when called will call the corresponding function in js.
I defined mine in the class JSInterop.cs:

public class JSInterop  
{
   public static string OnLoaded() =>
      RegisteredFunction.Invoke<string>("BlazorVision.JSInterop.OnLoaded");
}

The reason this is needed is that this script-file is loaded before the blazor part has had time to load so the fileInput element is not in the DOM and cannot be accessed. Once the blazor stuff is loaded we can call the javascript using the OnAfterRender function override

protected override void OnAfterRender()  
{
        base.OnAfterRender();
        JSInterop.OnLoaded();
}

Once an image is selected the js sets the src on the image to the file, this will trigger the onload-event which is bound to FileAdded in c#

 <img id="imgPreview" style="max-width:400px;" onload="@FileAdded" />

This function will read the Image-string, convert it to a byte[] and send it off to Custom vision using the settings defined in Configuration.cs.
Make sure to use the url and the prediction-key from the Custom vision api under "If you have an image file"

  public static class Configuration
    {
        public static string CustomVisionUrl => "YOUR-CUSTOM-VISION-URL";
        public static string PredictionKey => "YOUR-CUSTOM-VISION-PREDICTION-KEY";
        public static string ContentType => "application/octet-stream";
    }

There are some StateHasChanged()-calls in the FileAdded function to trigger updates to the UI.
Once the result comes back it is deserialized to a Predictions object and if any prediction result is over 75% probable it is set as the Prediction (which I generated using the brilliant Paste JSON as classes shortcut in VS 2017) to show in the H3 beneath the image.

  <h3>@PredictionResult?.Result</h3>

Success!

And of course it might fail :)

This was a fun way to spend a rainy Sunday, try it out and see how it works!