Azure Service Fabric: Service Endpoint resolution

Last week I spend a lot of time implementing something I thought would be easy, but took me a while in the end. Inside the Azure Service Fabric I have an actor that needs to talk to a web api hosted in a stateless service. To make this work I need to know the web api endpoint uri to call. My first attempt was to use the default HttpClient but this is not such a good idea as the service instances can move between nodes and the uri can change.

So what is the correct way to do this?

First thing we need to do is to create a communication client and implement the ICommunicationClient interface:

	public class HttpCommunicationClient : ICommunicationClient
	{
	public HttpCommunicationClient(HttpClient client, string address)
	{
	this.HttpClient = client;
	this.Url = new Uri(address);
	}
	public HttpClient HttpClient { get; }
	public Uri Url { get; }
	ResolvedServiceEndpoint ICommunicationClient.Endpoint { get; set; }
	string ICommunicationClient.ListenerName { get; set; }
	ResolvedServicePartition ICommunicationClient.ResolvedServicePartition { get; set; }
	}

view raw HttpCommunicationClient.cs hosted with ❤ by GitHub

This communication client will be created using a communication client factory. The Reliable Services API provides a CommunicationClientFactoryBase<TCommunicationClient>. This class implements a typical fault-handling retry pattern that makes retrying connections to resolved service endpoints easier.

We have to implement this abstract CommunicationClientFactoryBase class to handle logic that is specific to our communication stack:

	public class HttpCommunicationClientFactory : CommunicationClientFactoryBase<HttpCommunicationClient>
	{
	private HttpClient httpClient = new HttpClient();
	public HttpCommunicationClientFactory(IServicePartitionResolver resolver = null, IEnumerable<IExceptionHandler> exceptionHandlers = null)
	: base(resolver, CreateExceptionHandlers(exceptionHandlers))
	{
	}
	protected override void AbortClient(HttpCommunicationClient client)
	{
	// client with persistent connections should be abort their connections here.
	// HTTP clients don't hold persistent connections, so no action is taken.
	}
	protected override Task<HttpCommunicationClient> CreateClientAsync(string endpoint, CancellationToken cancellationToken)
	{
	// clients that maintain persistent connections to a service should
	// create that connection here.
	// an HTTP client doesn't maintain a persistent connection.
	return Task.FromResult(new HttpCommunicationClient(this.httpClient, endpoint));
	}
	protected override bool ValidateClient(HttpCommunicationClient client)
	{
	// client with persistent connections should be validated here.
	// HTTP clients don't hold persistent connections, so no validation needs to be done.
	return true;
	}
	protected override bool ValidateClient(string endpoint, HttpCommunicationClient client)
	{
	// client with persistent connections should be validated here.
	// HTTP clients don't hold persistent connections, so no validation needs to be done.
	return true;
	}
	private static IEnumerable<IExceptionHandler> CreateExceptionHandlers(IEnumerable<IExceptionHandler> additionalHandlers)
	{
	return new[] { new HttpExceptionHandler() }.Union(additionalHandlers ?? Enumerable.Empty<IExceptionHandler>());
	}
	}

view raw HttpCommunicationClientFactory.cs hosted with ❤ by GitHub

Finally, an exception handler is reponsible for determining what action to take when an exception occurs:

	public class HttpExceptionHandler : IExceptionHandler
	{
	public bool TryHandleException(ExceptionInformation exceptionInformation, OperationRetrySettings retrySettings, out ExceptionHandlingResult result)
	{
	if (exceptionInformation.Exception is TimeoutException)
	{
	result = new ExceptionHandlingRetryResult(exceptionInformation.Exception, false, retrySettings, retrySettings.DefaultMaxRetryCount);
	return true;
	}
	else if (exceptionInformation.Exception is ProtocolViolationException)
	{
	result = new ExceptionHandlingThrowResult();
	return true;
	}
	else if (exceptionInformation.Exception is SocketException)
	{
	result = new ExceptionHandlingRetryResult(exceptionInformation.Exception, false, retrySettings, retrySettings.DefaultMaxRetryCount);
	return true;
	}
	WebException we = exceptionInformation.Exception as WebException;
	if (we == null)
	{
	we = exceptionInformation.Exception.InnerException as WebException;
	}
	if (we != null)
	{
	HttpWebResponse errorResponse = we.Response as HttpWebResponse;
	if (we.Status == WebExceptionStatus.ProtocolError)
	{
	if (errorResponse.StatusCode == HttpStatusCode.NotFound)
	{
	// This could either mean we requested an endpoint that does not exist in the service API (a user error)
	// or the address that was resolved by fabric client is stale (transient runtime error) in which we should re-resolve.
	result = new ExceptionHandlingRetryResult(exceptionInformation.Exception, false, retrySettings, retrySettings.DefaultMaxRetryCount);
	return true;
	}
	if (errorResponse.StatusCode == HttpStatusCode.InternalServerError)
	{
	// The address is correct, but the server processing failed.
	// Retry the operation without re-resolving the address.
	result = new ExceptionHandlingRetryResult(exceptionInformation.Exception, true, retrySettings, retrySettings.DefaultMaxRetryCount);
	return true;
	}
	}
	if (we.Status == WebExceptionStatus.Timeout ||
	we.Status == WebExceptionStatus.RequestCanceled ||
	we.Status == WebExceptionStatus.ConnectionClosed ||
	we.Status == WebExceptionStatus.ConnectFailure)
	{
	result = new ExceptionHandlingRetryResult(exceptionInformation.Exception, false, retrySettings, retrySettings.DefaultMaxRetryCount);
	return true;
	}
	}
	result = null;
	return false;
	}
	}

view raw HttpExceptionHandler.cs hosted with ❤ by GitHub

Now we can call our Web API endpoint in a safe way:

	ServicePartitionClient<HttpCommunicationClient> partitionClient
	= new ServicePartitionClient<HttpCommunicationClient>(communicationFactory, serviceUri);
	var result = await partitionClient.InvokeWithRetryAsync(
	async (client) =>
	{
	return await client.HttpClient.GetAsync(new Uri(client.Url, $"/api/reports/full?fromdate={fromDate}&todate={toDate}"));
	});
	var jsonString = await result.Content.ReadAsStringAsync();

view raw callservice.cs hosted with ❤ by GitHub