CodingOnWheels.com

C# Functional Programming In-Depth (14) Asynchronous Function

Tue, 04 Mar 2025 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

Asynchronous function can improve the responsiveness and scalability of the application and service. C# 5.0 introduces asynchronous function to greatly simplify the async programming model.

Task, Task<TResult> and asynchrony

The C# async programming model uses System.Threading.Tasks.Task to represent async operation without output, and uses System.Threading.Tasks.Task<TResult> to represent async operation with TResult output:

namespace System.Threading.Tasks

{
public partial class Task : IAsyncResult
{
public Task(Action action); // Wraps () –> void function.

public void Start();

public void Wait();

public TaskStatus Status { get; } // Created, WaitingForActivation, WaitingToRun, Running, WaitingForChildrenToComplete, RanToCompletion, Canceled, Faulted.

public bool IsCanceled { get; }

public bool IsCompleted { get; }

public bool IsFaulted { get; }

public AggregateException Exception { get; }

Task ContinueWith(Action<Task> continuationAction);

Task<TResult> ContinueWith<TResult>(Func<Task, TResult> continuationFunction);

// Other members.
}

public partial class Task<TResult> : Task
{
public Task(Func<TResult> function); // Wraps () –> TResult function.

public TResult Result { get; }

public Task ContinueWith(Action<Task<TResult>> continuationAction);

public Task<TNewResult> ContinueWith<TNewResult>(
Func<Task<TResult>, TNewResult> continuationFunction);

// Other members.
}

}

Task can be constructed with () –>void function, and Task<TResult> can be constructed with () –> TResult function. They can be started by calling the Start method. A () –> void function or () –> TResult function runs synchronously on the thread. In contrast, a task runs asynchronously, and does not block the current thread. Its status can be queried by the Status, IsCanceled, IsCompleted, IsFaulted properties. A task can be waited by calling its Wait method, which blocks the current thread until the task is completed successfully, or fails, or is cancelled. For Task<TResult>, when the underlying async operation is completed successfully, the result is available through Result property. For Task or Task<TResult>, if the underlying async operation fails with exception, the exception is available through the Exception property. A task can be chained with another async continuation operation by calling the ContinueWith methods. When the task finishes running, the specified continuation starts running asynchronously. If the task already finishes running when its ContinueWith method is called, then the specified continuation immediately starts running. The following example constructs and starts a task to read a file, and chains another continuation task to write the contents to another file:

internal static partial class Functions

{
internal static void ConstructTask(string readPath, string writePath)
{
Thread.CurrentThread.ManagedThreadId.WriteLine(); // 10
Task<string> task = new Task<string>(() =>
{
Thread.CurrentThread.ManagedThreadId.WriteLine(); // 8
return File.ReadAllText(readPath);
});
task.Start();
Task continuationTask = task.ContinueWith(antecedentTask =>
{
Thread.CurrentThread.ManagedThreadId.WriteLine(); // 9
object.ReferenceEquals(antecedentTask, task).WriteLine(); // True
if (antecedentTask.IsFaulted)
{
antecedentTask.Exception.WriteLine();
}
else
{
File.WriteAllText(writePath, antecedentTask.Result);
}
});
continuationTask.Wait();
}

}

As async operations, when tasks are started, the wrapped functions are by default scheduled to runtime’s thread pool to execute, so that their thread ids are different from the caller thread id.

In .NET Framework, Task also implements System.Threading.IThreadPoolWorkItem and IDisposable interfaces. Its usage is the same as in .NET Core. Do not bother disposing tasks.

Task also provides Run methods to construct and automatically start tasks:

namespace System.Threading.Tasks

{
public partial class Task : IAsyncResult
{
public static Task Run(Action action);

public static Task<TResult> Run<TResult>(Func<TResult> function);
}

}

Now compare the following functions:

internal static void Write(string path, string contents) => File.WriteAllText(path, contents);

internal static Task<string>ReadAsync(string path) => Task.Run(() => File.ReadAllText(path));

Write without output and Read with string output run synchronously. WriteAsync with Task output and ReadAsync with Task<string> output run asynchronously, where Task can be viewed as future void, and Task<TResult> can be viewed as future TResult result. Here WriteAsync and ReadAsync become async by simply offloading the operations to thread pool. This is for demonstration purpose, and does not bring any scalability improvement. A better implementation is discussed later.

internal static void CallReadWrite(string path, string contents)

{
Write(path, contents); // Blocking.
// When the underlying write operation is done, the call is completed with no result.
string result = Read(path); // Blocking.
// When the underlying read operation is done, the call is completed with result available.

Task writeTask = WriteAsync(path, contents); // Non-blocking.
// When the task is constructed and started, the call is completed immediately.
// The underlying write operation is scheduled, and will be completed in the future with no result.
Task<string> readTask = ReadAsync(path); // Non-blocking.
// When the task is constructed and started, the call is completed immediately.
// The underlying read operation is scheduled, and will be completed in the future with result available.

}

When Write is called, its execution blocks the current thread. When the writing operation is done synchronously, it does not output any result, and then the caller thread can continue execution. Similarly, when Read is called, its execution blocks the current thread too. When the reading operation is done synchronously, it outputs the result, so that the result is available to the caller and the caller can continue execution. When WriteAsync is called, it calls Task.Run to construct a Task instance with the writing operation, start the task, then immediately outputs the task to the caller. Then the caller can continue without being blocked by the writing operation execution. By default, the writing operation is scheduled to thread pool, when it is done, the writing operation outputs no result, and the task’s Status is updated. Similarly, when ReadAsync is called, it also calls Task.Run to construct a Task<string> instance with the reading operation, start the task, then immediately outputs the task to the caller. Then the caller can continue without being blocked by the reading operation execution. By default, the reading operation is also scheduled to thread pool, when it is done, the reading operation has a result, and the task’s Status is updated, with the result available through the Result property.

Named async function

By default, named async function’s output type is Task or Task<TResult>, and has an Async or AsyncTask postfix in the name as the convention. The following example is a file read and write workflow of sync function calls:

internal static void ReadWrite(string readPath, string writePath)

{
string contents = Read(readPath);
Write(writePath, contents);

}

The same logic can be implemented by calling the async version of functions:

internal static async Task ReadWriteAsync(string readPath, string writePath)

{
string contents = await ReadAsync(readPath);
await WriteAsync(writePath, contents);

}

Here await keyword is used for each async function call, and the code structure remains the same as the sync workflow. When await keyword is used in function body, the async modifier is required for that function’s signature. Regarding the workflow outputs no result, the async function outputs Task (future void). This ReadWriteAsync function calls async functions, itself is also async function, since it has the async modifier and Task output. When ReadWriteAsync is called, it works the same way as ReadAsync and WriteAsync. it does not block its caller, and immediately outputs a task to represent the scheduled read and write workflow.

So the await keyword can be viewed as virtually waiting for the task’s underlying async operation to finish. If the task fails, exception is thrown. If the task is completed successfully, the continuation right after the await expression is called back. If the task has a result, await expression can extract the result. Therefore, the async workflow keeps the same looking of sync workflow. There is no ContinueWith call needed to build the continuation. The following example is a more complex database query workflow of sync function calls, with an int value as the query result:

internal static int Query(DbConnection connection, StreamWriter logWriter) // Output int.

{
try
{
connection.Open(); // Output void.
using (DbCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT 1;";
using (DbDataReader reader = command.ExecuteReader()) // Output DbDataReader.
{
if (reader.Read()) // Output bool.
{
return (int)reader[0];
}
throw new InvalidOperationException("Failed to call sync functions.");
}
}
}
catch (SqlException exception)
{
logWriter.WriteLine(exception.ToString()); // Output void.
throw new InvalidOperationException("Failed to call sync functions.", exception);
}

}

Here the DbConnection.Open, DbCommand.ExecuteReader, DbDataReader.Read, StreamWriter.WriteLine functions have async version provided as DbConnection.OpenAsync, DbCommand.ExecuteReaderAsync, DbDataReader.ReadAsync, StreamWriter.WriteLineAsync. They output either Task or Task<TResult>. With the async and await keywords, it easy to call these async functions:

internal static async Task<int> QueryAsync(

DbConnection connection, StreamWriter logWriter) // Output Task<int> instead of int.
{
try
{
await connection.OpenAsync(); // Output Task instead of void.
using (DbCommand command = connection.CreateCommand())
{
command.CommandText = "SELECT 1;";
using (DbDataReader reader = await command.ExecuteReaderAsync()) // Output Task<DbDataReader> instead of DbDataReader.
{
if (await reader.ReadAsync()) // Output Task<bool> instead of bool.
{
return (int)reader[0];
}
throw new InvalidOperationException("Failed to call async functions.");
}
}
}
catch (SqlException exception)
{
await logWriter.WriteLineAsync(exception.ToString()); // Output Task instead of void.
throw new InvalidOperationException("Failed to call async functions.", exception);
}

}

Again, the async workflow persists the same code structure as the sync workflow, including try-catch, using, if statements, etc. Without this syntax, it is a lot more complex to call ContinueWith and manually build above workflow. Regarding the async function has an int result, its output type is Task<int> (future int).

The above Write and Read functions calls File.WriteAllText and File.ReadAllText to execute sync I/O operation, which are internally implemented by calling StreamWriter.Write and StreamReader.ReadToEnd. Now with the async and await keywords, WriteAsync and ReadAsync can be reimplemented as real async I/O (assuming async I/O is actually supported by the underlying operating system) by calling StreamWriter.WriteAsync and StreamReader.ReadToEndAsync:

internal static async Task WriteAsync(string path, string contents)

{
// File.WriteAllText implementation:
// using (StreamWriter writer = new StreamWriter(new FileStream(
// path: path, mode: FileMode.Create, access: FileAccess.Write,
// share: FileShare.Read, bufferSize: 4096, useAsync: false)))
// {
// writer.Write(contents);
// }
using (StreamWriter writer = new StreamWriter(new FileStream(
path: path, mode: FileMode.Create, access: FileAccess.Write,
share: FileShare.Read, bufferSize: 4096, useAsync: true)))
{
await writer.WriteAsync(contents);
}
}

internal static async Task<string>ReadAsync(string path)
{
// File.ReadAllText implementation:
// using (StreamReader reader = new StreamReader(new FileStream(
// path: path, mode: FileMode.Open, access: FileAccess.Read,
// share: FileShare.Read, bufferSize: 4096, useAsync: false)))
// {
// return reader.ReadToEnd();
// }
using (StreamReader reader = new StreamReader(new FileStream(
path: path, mode: FileMode.Open, access: FileAccess.Read,
share: FileShare.Read, bufferSize: 4096, useAsync: true)))
{
return await reader.ReadToEndAsync();
}

}

There is one special scenario where async function has void output – async function as event handler. For example, ObservableCollection<T> has a CollectionChanged event:

namespace System.Collections.ObjectModel

{
public class ObservableCollection<T> : Collection<T>, INotifyCollectionChanged, INotifyPropertyChanged
{
public event NotifyCollectionChangedEventHandler CollectionChanged;

// Other members.
}
}

namespace System.Collections.Specialized
{
// (object, NotifyCollectionChangedEventArgs) –> void.
public delegate void NotifyCollectionChangedEventHandler(
object sender, NotifyCollectionChangedEventArgs e);

}

This event requires its handler to be a function of type (object, NotifyCollectionChangedEventArgs) –> void. So, when defining an async function as the above event’s handler, that async function has void output instead of Task:

private static readonly StringBuilder Logs = new StringBuilder();

}

The await keyword works with task from async function as well as any Task and Task<TResult> instance:

internal static async Task AwaitTasks(string path)

{
Task<string> task1 = ReadAsync(path);
string contents = await task1;
// Equivalent to: string contents = await ReadAsync(path);

Task task2 = WriteAsync(path, contents);
await task2;
// Equivalent to: await WriteAsync(path, contents);

Task task3 = Task.Run(() => { });
await task3;
// Equivalent to: await Task.Run(() => { });

Task<int> task4 = Task.Run(() => 0);
int result = await task4;
// Equivalent to: int result = await Task.Run(() => 0);

Task task5 = Task.Delay(TimeSpan.FromSeconds(10));
await task5;
// Equivalent to: await Task.Delay(TimeSpan.FromSeconds(10));

Task<int> task6 = Task.FromResult(result);
result = await task6;
// Equivalent to: result = await Task.FromResult(result);

}

If a task is never started, apparently it never finishes running. The code after its await expression is never called back:

internal static async Task HotColdTasks(string path)

{
Task hotTask = new Task(() => { });
hotTask.Start();
await hotTask;
hotTask.Status.WriteLine();

Task coldTask = new Task(() => { });
await coldTask;
coldTask.Status.WriteLine(); // Never execute.

}

Task not started yet is called cold task, and task already started is called hot task. As a convention, any function with task output should always output a hot task. All APIs in .NET Standard follow this convention.

Awaitable-awaiter pattern

C# compiles the await expression with the awaitable-awaiter pattern. Besides Task and Task<TResult>, the await keyword can be used with any awaitable type. An awaitable type has a GetAwaiter instance or extension method to output an awaiter. An awaiter type implements System.Runtime.CompilerServices.INotifyCompletion interface, also has an IsCompleted property with bool output, and a GetResult instance method with or without output. The following IAwaitable and IAwaiter interfaces demonstrate the awaitable-awaiter pattern for operation with no result:

public interface IAwaitable

{
IAwaiter GetAwaiter();
}

public interface IAwaiter : INotifyCompletion
{
bool IsCompleted { get; }

void GetResult(); // No result.

}

And the following IAwaitable<TResult> and IAwaiter<TResult> interfaces demonstrate the awaitable-awaiter pattern for operations with a result:

public interface IAwaitable<TResult>

{
IAwaiter<TResult>GetAwaiter();
}

public interface IAwaiter<TResult>: INotifyCompletion
{
bool IsCompleted { get; }

TResult GetResult(); // TResult result.

}

And INotifyCompletion interface has a single OnCompleted method to chain a continuation:

namespace System.Runtime.CompilerServices

{
public interface INotifyCompletion
{
void OnCompleted(Action continuation);
}

}

Here is how Task and Task<TResult> implement the awaitable-awaiter pattern. Task can be virtually viewed as implementation of IAwaitable, it has a GetAwaiter instance method outputting System.Runtime.CompilerServices.TaskAwaiter, which can be virtually viewed as implementation of IAwaiter; Similarly, Task<TResult> can be virtually viewed as implementation of IAwaitable<TResult>, it has a GetAwaiter method outputting System.Runtime.CompilerServices.TaskAwaiter<TResult>, which can be virtually viewed as implementation of IAwaiter<TResult>:

namespace System.Threading.Tasks

{
public partial class Task : IAsyncResult
{
public TaskAwaiter GetAwaiter();
}

public partial class Task<TResult> : Task
{
public TaskAwaiter<TResult> GetAwaiter();
}
}

namespace System.Runtime.CompilerServices
{
public struct TaskAwaiter : ICriticalNotifyCompletion, INotifyCompletion
{
public bool IsCompleted { get; }

public void GetResult(); // No result.

public void OnCompleted(Action continuation);

// Other members.
}

public struct TaskAwaiter<TResult> : ICriticalNotifyCompletion, INotifyCompletion
{
public bool IsCompleted { get; }

public TResult GetResult(); // TResult result.

public void OnCompleted(Action continuation);

// Other members.
}

}

Any other type can be used with the await keyword, as long as the awaitable-awaiter pattern is implemented. Take delegate type Action as example, a GetAwaiter method can be easily implemented as its extension method, by reusing above TaskAwaiter:

public static TaskAwaiter GetAwaiter(this Action action) => Task.Run(action).GetAwaiter();

Similarly, this pattern can be implemented for Func<TResult>, by reusing TaskAwaiter<TResult>:

public static TaskAwaiter<TResult> GetAwaiter<TResult>(this Func<TResult> function) =>

Task.Run(function).GetAwaiter();

Now the await keyword can be directly used with a function of Action type or Func<TResult> type:

internal static async Task AwaitFunctions(string readPath, string writePath)

{
Func<string>read = () => File.ReadAllText(readPath);
string contents = await read;

Action write = () => File.WriteAllText(writePath, contents);
await write;

}

Async state machine

As fore mentioned, with async and await keywords, an async function outputs a task immediately, so it is non-blocking. At compile time, the workflow of an async function is compiled to an async state machine. At runtime, when this async function is called, it just starts that generated async state machine , and immediately outputs a task representing the workflow in the async state machine. To demonstrate this, define the following async methods:

internal static async Task<T> Async<T>(T value)

{
T value1 = Start(value);
T result1 = await Async1(value1);
T value2 = Continuation1(result1);
T result2 = await Async2(value2);
T value3 = Continuation2(result2);
T result3 = await Async3(value3);
T result = Continuation3(result3);
return result;
}

internal static T Start<T>(T value) => value;

internal static Task<T> Async1<T>(T value) => Task.Run(() => value);

internal static T Continuation1<T>(T value) => value;

internal static Task<T> Async2<T>(T value) => Task.FromResult(value);

internal static T Continuation2<T>(T value) => value;

internal static Task<T> Async3<T>(T value) => Task.Run(() => value);

internal static T Continuation3<T>(T value) => value;

After compilation, the async modifier is gone. The async function becomes a normal function to start an async state machine:

[AsyncStateMachine(typeof(AsyncStateMachine<>))]

internal static Task<T> CompiledAsync<T>(T value)
{
AsyncStateMachine<T>asyncStateMachine = new AsyncStateMachine<T>()
{
Value = value,
Builder = AsyncTaskMethodBuilder<T>.Create(),
State = -1 // -1 means start.
};
asyncStateMachine.Builder.Start(ref asyncStateMachine);
return asyncStateMachine.Builder.Task;

}

And the generated async state machine is a structure in release build, and a class in debug build:

[CompilerGenerated]

[StructLayout(LayoutKind.Auto)]
private struct AsyncStateMachine<TResult>: IAsyncStateMachine
{
public int State;

public AsyncTaskMethodBuilder<TResult> Builder;

public TResult Value;

private TaskAwaiter<TResult> awaiter;

void IAsyncStateMachine.MoveNext()
{
TResult result;
try
{
switch (this.State)
{
case -1: // Start code from the beginning to the 1st await.
// Workflow begins.
TResult value1 = Start(this.Value);
this.awaiter = Async1(value1).GetAwaiter();
if (this.awaiter.IsCompleted)
{
// If the task returned by Async1 is already completed, immediately execute the continuation.
goto case 0;
}
else
{
this.State = 0;
// If the task returned by Async1 is not completed, specify the continuation as its callback.
this.Builder.AwaitUnsafeOnCompleted(ref this.awaiter, ref this);
// Later when the task returned by Async1 is completed, it calls back MoveNext, where State is 0.
return;
}
case 0: // Continuation code from after the 1st await to the 2nd await.
// The task returned by Async1 is completed. The result is available immediately through GetResult.
TResult result1 = this.awaiter.GetResult();
TResult value2 = Continuation1(result1);
this.awaiter = Async2(value2).GetAwaiter();
if (this.awaiter.IsCompleted)
{
// If the task returned by Async2 is already completed, immediately execute the continuation.
goto case 1;
}
else
{
this.State = 1;
// If the task returned by Async2 is not completed, specify the continuation as its callback.
this.Builder.AwaitUnsafeOnCompleted(ref this.awaiter, ref this);
// Later when the task returned by Async2 is completed, it calls back MoveNext, where State is 1.
return;
}
case 1: // Continuation code from after the 2nd await to the 3rd await.
// The task returned by Async2 is completed. The result is available immediately through GetResult.
TResult result2 = this.awaiter.GetResult();
TResult value3 = Continuation2(result2);
this.awaiter = Async3(value3).GetAwaiter();
if (this.awaiter.IsCompleted)
{
// If the task returned by Async3 is already completed, immediately execute the continuation.
goto case 2;
}
else
{
this.State = 2;
// If the task returned by Async3 is not completed, specify the continuation as its callback.
this.Builder.AwaitUnsafeOnCompleted(ref this.awaiter, ref this);
// Later when the task returned by Async3 is completed, it calls back MoveNext, where State is 1.
return;
}
case 2: // Continuation code from after the 3rd await to the end.
// The task returned by Async3 is completed. The result is available immediately through GetResult.
TResult result3 = this.awaiter.GetResult();
result = Continuation3(result3);
this.State = -2; // -2 means end.
this.Builder.SetResult(result);
// Workflow ends.
return;
}
}
catch (Exception exception)
{
this.State = -2; // -2 means end.
this.Builder.SetException(exception);
}
}

[DebuggerHidden]
void IAsyncStateMachine.SetStateMachine(IAsyncStateMachine asyncStateMachine) =>
this.Builder.SetStateMachine(asyncStateMachine);

}

The generated async state machine is a finite state machine:

The workflow is compiled into a switch statement in its MoveNext method. The original workflow is split to 4 parts by the 3 await keywords, and compiled to 4 cases of the switch statement. The parameter of the workflow is compiled as a field of the state machine, so it can be accessed by the workflow inside MoveNext. When the state machine is initialized, its initial state is –1, which means to start. Once the state machine is started, MoveNext is called, and the case –1 block is executed, which has the code from the beginning of the workflow to the first await expression, which is compiled to a GetAwaiter call. If the awaiter is already completed, then the continuation should immediate be executed by going to case 0 block directly; If the awaiter is not completed, the continuation (MoveNext call with next state 0) is specified as the awaiter’s callback when it is completed in the future. In either case, when code in case 0 block is executed, the previous awaiter is already completed, and its result is immediately available through its GetResult method. The execution goes on in the same pattern, until the last block of case 2 is executed.

Runtime context capture

At runtime, when executing an await expression, if the awaited task is not completed yet, the continuation is scheduled as callback. As a result, the continuation can be executed by a thread different from initial caller thread. By default, the initial thread’s runtime context information is captured, and are reused by the callback thread to execute the continuation. To demonstrate this, the above awaitable-awaiter pattern for Action can be re-implemented with the following custom awaiter:

public static IAwaiter GetAwaiter(this Action action) =>

new ActionAwaiter(Task.Run(action));

public class ActionAwaiter : IAwaiter
{
private readonly (SynchronizationContext, TaskScheduler, ExecutionContext) runtimeContext;

private readonly Task task;

public ActionAwaiter(Task task) =>
(this.task, this.runtimeContext) = (task, RuntimeContext.Capture()); // Capture runtime context when initialized.

public bool IsCompleted => this.task.IsCompleted;

public void GetResult() => this.task.Wait();

public void OnCompleted(Action continuation) => this.task.ContinueWith(task =>
this.runtimeContext.Execute(continuation));// Execute continuation on captured runtime context.

}

When the awaiter is constructed, it captures the runtime context information, including System.Threading.SynchronizationContext, System.Threading.Tasks.TaskScheduler, and System.Threading.ExecutionContext of current thread. Then in OnCompleted, when the continuation is called back, it is executed with the previously captured runtime context information. The custom awaiter can be implemented for Func<TResult> in the same pattern:

public static IAwaiter<TResult> GetAwaiter<TResult>(this Func<TResult> function) =>

new FuncAwaiter<TResult>(Task.Run(function));

public class FuncAwaiter<TResult> : IAwaiter<TResult>
{
private readonly (SynchronizationContext, TaskScheduler, ExecutionContext) runtimeContext =
RuntimeContext.Capture();

private readonly Task<TResult> task;

public FuncAwaiter(Task<TResult> task) => (this.task, this.runtimeContext) = (task, RuntimeContext.Capture()); // Capture runtime context when initialized.

public bool IsCompleted => this.task.IsCompleted;

public TResult GetResult() => this.task.Result;

public void OnCompleted(Action continuation) => this.task.ContinueWith(task =>
this.runtimeContext.Execute(continuation)); // Execute continuation on captured runtime context.

}

The following is a basic implementation of runtime context capture and resume:

public static class RuntimeContext

{
public static (SynchronizationContext, TaskScheduler, ExecutionContext) Capture() =>
(SynchronizationContext.Current, TaskScheduler.Current, ExecutionContext.Capture());

public static void Execute(
this (SynchronizationContext, TaskScheduler, ExecutionContext) runtimeContext, Action continuation)
{
var (synchronizationContext, taskScheduler, executionContext) = runtimeContext;
if (synchronizationContext != null && synchronizationContext.GetType() != typeof(SynchronizationContext))
{
if (synchronizationContext == SynchronizationContext.Current)
{
executionContext.Run(continuation);
}
else
{
executionContext.Run(() => synchronizationContext.Post(
d: state => continuation(), state: null));
}
return;
}
if (taskScheduler != null && taskScheduler != TaskScheduler.Default)
{
Task continuationTask = new Task(continuation);
continuationTask.Start(taskScheduler);
return;
}
executionContext.Run(continuation);
}

private static void Run(this ExecutionContext executionContext, Action continuation)
{
if (executionContext != null)
{
ExecutionContext.Run(
executionContext: executionContext, callback: state => continuation(), state: null);
}
else
{
continuation();
}
}

}

When Capture is called, it captures a 3-tuple of SynchronizationContext, TaskScheduler, and ExecutionContext When the continuation is executed, first the previously captured SynchronizationContext is checked. If a specialized SynchronizationContext is captured and it is different from current SynchronizationContext, then the continuation is executed with the captured SynchronizationContext and ExecutionContext. When there is no specialized SynchronizationContext captured, then the TaskScheduler is checked. If a specialized TaskScheduler is captured, it is used to schedule the continuation as a task. For all the other cases, the continuation is executed with the captured ExecutionContext.

Task and Task<TResult> provides a ConfigureAwait method to specify whether the continuation is marshalled to the previously captured runtime context:

namespace System.Threading.Tasks

{
public partial class Task : IAsyncResult
{
public ConfiguredTaskAwaitable ConfigureAwait(bool continueOnCapturedContext);
}

public partial class Task<TResult> : Task
{
public ConfiguredTaskAwaitable<TResult> ConfigureAwait(bool continueOnCapturedContext);
}

}

To demonstrate the runtime context capture, define a custom task scheduler, which simply start a background thread to execute each task:

public class BackgroundThreadTaskScheduler : TaskScheduler

{
protected override IEnumerable<Task> GetScheduledTasks() =>
throw new NotImplementedException();

protected override void QueueTask(Task task) =>
new Thread(() => this.TryExecuteTask(task)) { IsBackground = true }.Start();

protected override bool TryExecuteTaskInline(
Task task, bool taskWasPreviouslyQueued) =>
this.TryExecuteTask(task);

}

The following async function has 2 await expressions, where ConfigureAwait is called with different bool values:

internal static async Task ConfigureRuntimeContextCapture(

string readPath, string writePath)
{
TaskScheduler taskScheduler1 = TaskScheduler.Current;
string contents = await ReadAsync(readPath).ConfigureAwait(
continueOnCapturedContext: true);
// Equivalent to: await ReadAsync(readPath);

// Continuation is executed with captured runtime context.
TaskScheduler taskScheduler2 = TaskScheduler.Current;
object.ReferenceEquals(taskScheduler1, taskScheduler2).WriteLine(); // True
await WriteAsync(writePath, contents).ConfigureAwait(
continueOnCapturedContext: false);

// Continuation is executed without captured runtime context.
TaskScheduler taskScheduler3 = TaskScheduler.Current;
object.ReferenceEquals(taskScheduler1, taskScheduler3).WriteLine(); // False

}

To demonstrate the task scheduler capture, call the above async function by specifying the custom task scheduler:

internal static async Task CallConfigureContextCapture(string readPath, string writePath)

{
Task<Task> task = new Task<Task>(() =>
ConfigureRuntimeContextCapture(readPath, writePath));
task.Start(new BackgroundThreadTaskScheduler());
await task.Unwrap(); // Equivalent to: await await task;

}

Here since async function ConfigureRuntimeContextCapture outputs Task, so the task constructed with () -> Task function is of type Task<Task>. Similarly, if task is constructed with () -> Task<TResult> function, the constructed task is of type Task<Task<TResult>>. For this scenario, Unwrap extension methods are provided to convert nested task to normal task:

namespace System.Threading.Tasks

{
public static class TaskExtensions
{
public static Task Unwrap(this Task<Task> task);

public static Task<TResult> Unwrap<TResult>(this Task<Task<TResult>> task);
}

}

When start executing ConfigureRuntimeContextCapture, the initial task scheduler is specified to be the custom task scheduler, BackgroundThreadTaskScheduler. In the first await expression, ConfigureAwait is called with true, so that the runtime context information is captured and the continuation is executed with the captured runtime context information. This is the default behaviour, so calling ConfigureAwait with true is equivalent to not calling ConfigureAwait at all. As a result, the first continuation is executed with the same custom task scheduler. In the second await expression, ConfigureAwait is called with false, so the runtime context information is not captured. As a result, the second continuation is executed with the default task scheduler, System.Threading.Tasks.ThreadPoolTaskScheduler.

The runtime context capture can be also demonstrated by SynchronizationContext. SynchronizationContext is inherited by different implementations in different application models, for example:

· ASP.NET: System.Web.AspNetSynchronizationContext

· WPF: System.Windows.Threading.DispatcherSynchronizationContext

· WinForms: System.Windows.Forms.WindowsFormsSynchronizationContext

· Windows Universal: System.Threading.WinRTCoreDispatcherBasedSynchronizationContext

· Xamarin.Android: Android.App.SyncContext

· Xamarin.iOS: UIKit.UIKitSynchronizationContext

Take Windows Universal application as example. In Visual Studio on Windows, create a Windows Universal application, add a button to its UI:

In the code behind, implement the Click event handler as an async function:

private async void ButtonClick(object sender, RoutedEventArgs e)

{
SynchronizationContext synchronizationContext1 = SynchronizationContext.Current;
ExecutionContext executionContext1 = ExecutionContext.Capture();
await Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(
continueOnCapturedContext: true);
// Equivalent to: await Task.Delay(TimeSpan.FromSeconds(1));

// Continuation is executed with captured runtime context.
SynchronizationContext synchronizationContext2 = SynchronizationContext.Current;
Debug.WriteLine(synchronizationContext1 == synchronizationContext2); // True
this.Button.Background = new SolidColorBrush(Colors.Blue); // UI update works.
await Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(
continueOnCapturedContext: false);

// Continuation is executed without captured runtime context.
SynchronizationContext synchronizationContext3 = SynchronizationContext.Current;
Debug.WriteLine(synchronizationContext1 == synchronizationContext3); // False
this.Button.Background = new SolidColorBrush(Colors.Yellow); // UI update fails.
// Exception: The application called an interface that was marshalled for a different thread.

}

For Windows Universal application, the SynchronizationContext is only available for the UI thread, and application UI can only be updated with UI thread’s SynchronizationContext. When the button is clicked, the UI thread calls the async function ButtonClick, so the UI thread’s SynchronizationContext is captured. Similar to the previous example, when ConfigureAwait is called with true, the continuation is executed with the previously captured SynchronizationContext, so the continuation can update the UI successfully. When ConfigureAwait is called with true, the continuation is not executed with the SynchronizationContext captured from UI thread, and it fails to update the UI and throws exception.

Generalized async function output type and async method builder

Since C# 7, async function is supported to have any awaitable output type, as long as it has an async method builder specified. Take Func<TResult> is already awaitable with the previously defined GetAwaiter extension method as example, it is already awaitable with the previously defined GetAwaiter extension method, and can be used in await expression just like task. However, it cannot be used with async modifier to be async function output type just like task. To make Func<TResult> output type of async function, the following FuncAwaitable<TResult> type is defined as a wrapper of Func<TResult>. This wrapper is an awaitable type, its GetAwaiter instance method reuses previously defined FuncAwater<TResult> as its awaiter:

[AsyncMethodBuilder(typeof(AsyncFuncAwaitableMethodBuilder<>))]

public class FuncAwaitable<TResult>: IAwaitable<TResult>
{
private readonly Func<TResult> function;

public FuncAwaitable(Func<TResult> function) => this.function = function;

public IAwaiter<TResult> GetAwaiter() =>
new FuncAwaiter<TResult>(Task.Run(this.function));

}

This wrapper type is associated with its async method builder with System.Runtime.CompilerServices.AsyncMethodBuilderAttribute. The async method builder for FuncAwaitable<TResult> is implemented as:

public class AsyncFuncAwaitableMethodBuilder<TResult>

{
private AsyncTaskMethodBuilder<TResult> taskMethodBuilder;

private TResult result;

private bool hasResult;

private bool useBuilder;

public static AsyncFuncAwaitableMethodBuilder<TResult> Create() =>
new AsyncFuncAwaitableMethodBuilder<TResult>()
{
taskMethodBuilder = AsyncTaskMethodBuilder<TResult>.Create()
};

public void Start<TStateMachine>(ref TStateMachine stateMachine)
where TStateMachine : IAsyncStateMachine =>
this.taskMethodBuilder.Start(ref stateMachine);

public void SetStateMachine(IAsyncStateMachine stateMachine) =>
this.taskMethodBuilder.SetStateMachine(stateMachine);

public void SetResult(TResult result)
{
if (this.useBuilder)
{
this.taskMethodBuilder.SetResult(result);
}
else
{
this.result = result;
this.hasResult = true;
}
}

public void SetException(Exception exception) =>
this.taskMethodBuilder.SetException(exception);

public FuncAwaitable<TResult> Task
{
get
{
if (this.hasResult)
{
TResult result = this.result;
return new FuncAwaitable<TResult>(() => result);
}

this.useBuilder = true;
Task<TResult>task = this.taskMethodBuilder.Task;
return new FuncAwaitable<TResult>(() => task.Result);
}
}

public void AwaitOnCompleted<TAwaiter, TStateMachine>(
ref TAwaiter awaiter, ref TStateMachine stateMachine)
where TAwaiter : INotifyCompletion where TStateMachine : IAsyncStateMachine
{
this.useBuilder = true;
this.taskMethodBuilder.AwaitOnCompleted(ref awaiter, ref stateMachine);
}

public void AwaitUnsafeOnCompleted<TAwaiter, TStateMachine>(
ref TAwaiter awaiter, ref TStateMachine stateMachine)
where TAwaiter : ICriticalNotifyCompletion where TStateMachine : IAsyncStateMachine
{
this.useBuilder = true;
this.taskMethodBuilder.AwaitUnsafeOnCompleted(ref awaiter, ref stateMachine);
}

}

Now the FuncAwaitable<TResult> type can be output type of async function, just like task:

internal static async FuncAwaitable<T> AsyncFunctionWithFuncAwaitable<T>(T value)

{
await Task.Delay(TimeSpan.FromSeconds(1));
return value;

}

Its compilation is in the same pattern as async function with task output. The only difference is, in the generated async state machine, the builder field become the specified AsyncFuncAwaitableMethodBuilder<TResult>, instead of the AsyncTaskMethodBuilder<TResult> for task. And apparently, this async function can be called in the await expression since its output type FuncAwaitable<TResult> is awaitable:

internal static async Task CallAsyncFunctionWithFuncAwaitable<T>(T value)

{
T result = await AsyncFunctionWithFuncAwaitable(value);

}

ValueTask<TResult> and performance

With the generalized async function output type support, Microsoft also provides a System.Threading.Tasks.ValueTask<TResult> awaitable structure in the System.Threading.Tasks.Extensions NuGet package:

namespace System.Threading.Tasks

{
[AsyncMethodBuilder(typeof(AsyncValueTaskMethodBuilder<>))]
[StructLayout(LayoutKind.Auto)]
public struct ValueTask<TResult> : IEquatable<ValueTask<TResult>>
{
public ValueTask(TResult result);

public ValueTask(Task<TResult> task);

public ValueTaskAwaiter<TResult> GetAwaiter();

// Other members.
}

}

Its awaiter is System.Threading.Tasks.ValueTaskAwaiter<TResult>, and its async method builder is System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder<TResult>, which are provided in the same NuGet package. So ValueTask<TResult> can be used with both await expression and async function. As a value type, it can be allocated and deallocated on stack, with better performance than reference type Task<TResult>. Also, unlike Task<TResult> as a wrapper of Func<TResult> operation, ValueTask<TResult> can be a wrapper of either Func<TResult> operation or TResult result. So ValueTask<TResult> can improve the performance for async function that may have result available before any async operation. The following example downloads data from the specified URI:

private static readonly Dictionary<string, byte[]> Cache =

new Dictionary<string, byte[]>(StringComparer.OrdinalIgnoreCase);

internal static async Task<byte[]> DownloadAsyncTask(string uri)
{
// All code compiled to async state machine. When URI is cached, async state machine is still started.
if (Cache.TryGetValue(uri, out byte[] cachedResult))
{
return cachedResult;
}
using (HttpClient httpClient = new HttpClient())
{
byte[] result = await httpClient.GetByteArrayAsync(uri);
Cache.Add(uri, result);
return result;
}

}

It first checks the cache, if the data is already cached for the specified URI, then it outputs the cached data directly, no async operation is needed. However, at compile time, since the function has the async modifier, the entire function body, including the if statement, is compiled into an async state machine. At runtime, a task is always allocated on the heap and should be garbage collected, and the async state machine is always started, even when the result is available in the cache and no async operation is needed. With ValueTask<TResult>, this can be easily optimized:

internal static ValueTask<byte[]> DownloadAsyncValueTask(string uri)

{
// Not compiled to async state machine. When URI is cached, no async state machine is started.
return Cache.TryGetValue(uri, out byte[] cachedResult)
? new ValueTask<byte[]>(cachedResult)
: new ValueTask<byte[]>(DownloadAsync());

async Task<byte[]> DownloadAsync()
{
// Compiled to async state machine. When URI is not cached, async state machine is started.
using (HttpClient httpClient = new HttpClient())
{
byte[] result = await httpClient.GetByteArrayAsync(uri);
Cache.Add(uri, result);
return result;
}
}

}

Now the function becomes a sync function with awaitable ValueTask<TResult> output. When the result is available in the cache, the async local function is not called, so there is no async operation or async state machine involved, and there is no task allocated on heap. The async operation is encapsulated in the async local function, which is compiled to async state machine, and is only involved when the result is not available in the cache. As a result, the performance can be improved, especially when the cache is hit frequently.

Anonymous async function

The async and await keywords can be used with the lambda expression syntax for anonymous function. Just like named async function, anonymous async function’s output type is task:

internal static async Task AsyncAnonymousFunction(string readPath, string writePath)

{
Func<string, Task<string>>readAsync = async path =>
{
using (StreamReader reader = new StreamReader(new FileStream(
path: path, mode: FileMode.Open, access: FileAccess.Read,
share: FileShare.Read, bufferSize: 4096, useAsync: true)))
{
return await reader.ReadToEndAsync();
}
};
Func<string, string, Task>writeAsync = async (path, contents) =>
{
using (StreamWriter writer = new StreamWriter(new FileStream(
path: path, mode: FileMode.Create, access: FileAccess.Write,
share: FileShare.Read, bufferSize: 4096, useAsync: true)))
{
await writer.WriteAsync(contents);
}
};

string result = await readAsync(readPath);
await writeAsync(writePath, result);

}

The above async lambda expressions are compiled as methods of closure class, in the same pattern as normal sync lambda expressions.

Since task can be constructed with anonymous function with any output type, it can be constructed with anonymous async function with task output too:

internal static async Task ConstructTaskWithAsyncAnonymousFunction(

string readPath, string writePath)
{
Task<Task<string>> task1 = new Task<Task<string>>(async () =>
await ReadAsync(readPath));
task1.Start(); // Cold task needs to be started.
string contents = await task1.Unwrap(); // Equivalent to: string contents = await await task1;

Task<Task> task2 = new Task<Task>(async () => await WriteAsync(writePath, null));
task2.Start(); // Cold task needs to be started.
await task2.Unwrap(); // Equivalent to: await await task2;

}

The first task is constructed with anonymous async function of type () –> Task<string>, so the constructed task is of type Task<Task<string>>. Similarly, the second task is constructed with anonymous async function of type () –> Task, so the constructed task is of type Task<Task>. As fore mentioned, nested task can be unwrapped and awaited.

internal static async Task RunAsyncWithAutoUnwrap(string readPath, string writePath)

{
Task<string> task1 = Task.Run(async () => await ReadAsync(readPath)); // Automatically unwrapped.
string contents = await task1; // Hot task.

Task task2 = Task.Run(async () => await WriteAsync(writePath, contents)); // Automatically unwrapped.
await task2; // Hot task.

}

Asynchrnous sequence: IAsyncEnumerable<T>

In C# 8.0, an async function can return a sequence of values using the return type IAsyncEnumerable<T>:

namespace System.Collections.Generic

{
public interface IAsyncEnumerable<out T>
{
IAsyncEnumerator<T> GetAsyncEnumerator(CancellationToken cancellationToken = default);
}

public interface IAsyncEnumerator<out T> : IAsyncDisposable
{
T Current { get; }

ValueTask<bool> MoveNextAsync();
}

}

The following example define a async function that generates a sequence of values by using yield statement:

internal static async IAsyncEnumerable<string>DownloadAsync(IEnumerable<Uri> uris)

{
using WebClient webClient = new WebClient();
foreach (Uri uri in uris)
{
string webPage = await webClient.DownloadStringTaskAsync(uri);
yield return webPage;
}

}

The following example consumes an async sequence by pulling the values asynchronously using async foreach statement:

internal static async void PrintDownloadAsync(IEnumerable<Uri> uris)

{
IAsyncEnumerable<string> webPages = DownloadAsync(uris);
await foreach (string webPage in webPages)
{
Trace.WriteLine(webPage);
}

}

In above 2 async functions, the code are both compiled to a state macine similar to async functions returning a task.

async using declaration: IAsyncDispose

In C# 8.0, an instance can be decared with async using keywords, if its type implements System.IAsyncDispose interface:

namespace System

{
public interface IAsyncDisposable
{
ValueTask DisposeAsync();
}

}

For example, FileStream type implements IAsyncDispose:

internalstaticasyncvoid AsyncUsing(string file)

{
awaitusing FileStream fileStream = File.OpenRead(file);
Trace.WriteLine(fileStream.Length);

}

The code in the above async function is also compiled to a state machine.

Summary

C# supports async function as a simplified asynchronous programming model. In C#, async operation is represented by Task or Task<TResult>. A task can be async function output type, and can be used in await expression. Besides task, any type following the awaitable-awaiter pattern can be used in await expression. Async function is compiled to async state machine. It can capture the caller’s runtime context, and execute the continuation code with the captured context. C# generalizes async function output type with async method builder support, and a ValueTask<TResult> type is provided to improve async function performance. C# also support async local function and async anonymous function.

Functional Programming and LINQ via C#

Sun, 02 Mar 2025 00:00:00 GMT

Acclaim
Contents at a Glance
Table of Contents

This is a book on functional programming and LINQ programming via C# language. It discusses:

Functional programming via C# in-depth
Use functional LINQ to work with local data and cloud data
The underlying mathematics theories of functional programming and LINQ, including Lambda Calculus and Category Theory

Acclaim

Microsoft:

“An excellent book for those of us who need to get in-depth understanding on LINQ and functional programming with latest C# language. The author made sure this book includes the latest and cross-platform knowledge for the language, the framework, as well as the underlying mathematical theories.” Hongfei Guo Partner Group Engineering Manager at Microsoft
“This book explains practical and in-depth material clearly, concisely and accurately to the areas of the C# language, functional programming, and LINQ on .NET Framework and .NET Core. This is a great book for anyone wanting to understand the whys and hows behind these important technologies.” Samer Boshra Principal Software Development Engineer at Microsoft
“This is a great book for developers who want to go functional programming. It's one-stop shopping for serious developers who have to get up to speed with LINQ and functional programming quickly and in-depth. I'll keep this book on my desk not on my bookshelf.” Roshan Kommusetty Principal Software Engineering Manager at Microsoft
“This is a great book for C# developers, it covers both basic C# programming concepts for the beginners new to the .NET world, and C# advanced constructs for experienced .NET programmers. The book is up to date, talks C# 7.0 new language features and demonstrates how you can use them for functional programming. Thanks for the awesome work!” Mark Zhou Principal Software Engineering Manager at Microsoft
“I like the way the author presented the detailed knowledge with a lot of examples. As a data scientist with statistics background in a number of industries, I can pick up C# programming and LINQ quickly when I followed the book. The book was concise and easy to read. It was a pleasant experience for me to spend my time emerging myself in the book in the sunshine weekday afternoon.” Xue Liu Senior Data Scientist at Microsoft
“Functional Programming and LINQ in C# language, have been fully and clearly unraveled in this book, with many practical examples. The author has not saved any effort to go beyond scratching the surface of C# language and has successfully explained the magic behind the scene. This book is a must-have for anyone who wants to understand functional programming using C#.” Jie Mei Data & Applied Scientist at Microsoft

Academy and more:

“This book provides comprehensive and in-depth information about the C# functional programming and LINQ technologies to application developers on both .NET Framework and .NET Core. The detailed text and wealth of examples will give a developer a clear and solid understanding of C# language, functional programming and using LINQ to work with different data domains.” Dong Si Assistant Professor, Department of Computer Science, University of Washington, Bothell
“This book offers a comprehensive, in-depth, yet easy-to-understand tutorial to functional C# programming and LINQ. Filled with detailed explanations and real-world examples, this book is highly valuable for beginners and experienced developers alike.” Shuang Zhao Assistant Professor, Department of Computer Science, University of California, Irvine
“This excellent book is an in-depth and also readable exploration of C# functional programming and LINQ programming. It covers .NET Framework and .NET Core in great detail.” Yang Sha Engineering Manager at Google
“Great book! It takes a hands-on approach to LINQ and functional programming in an easy to understand format. I would highly recommend this book to developers looking to develop expertise in C#, functional programming, and LINQ.” Himanshu Lal Software Engineering Manager at Facebook
“This is a great book that combines practical examples with in-depth analysis of LINQ and functional programming in C#. Dixin leverages his expertise in .NET to provide a well written tutorial on the effective use of LINQ and an overview of the theoretical principles behind it. A must read for anyone working on these technologies!” Dimitrios Soulios Director at Goldman Sachs

Contents at a Glance

The contents are organized as the following chapters:

Part 1 Code - covers functional programming via C#, and fundamentals of LINQ.
- Chapter 1 Functional programming and LINQ paradigm
  - What is LINQ, how LINQ uses language to work with many different data domains.
  - Programming paradigm, imperative vs. declarative programming, object-oriented vs. functional programming.
- Chapter 2 Functional programming in depth
  - C# fundamentals for beginners.
  - Aspects of functional programming via C#, including function type, named/anonymous/local function, closure, lambda, higher-order function, currying, partial application, first class function, function composition, query expression, covariance/contravariance, immutability, tuple, purity, async function, pattern matching, etc., including how C# is processed at compile time and runtime.
Part 2 Data - covers how to use functional LINQ to work with different data domains in the real world, and how LINQ works internally.
- Chapter 3 LINQ to Objects
  - How to use functional LINQ queries to work with objects, covering all LINQ and Ix.
  - How the LINQ to Objects query methods are implemented, how to implement useful custom LINQ queries.
- Chapter 4 LINQ to XML
  - How to modeling XML data, and use functional LINQ queries to work with XML data.
  - How to use the other LINQ to XML APIs to manipulate XML data.
- Chapter 5 Parallel LINQ
  - How to use parallelized functional LINQ queries to work with objects.
  - Performance analysis for parallel/sequential LINQ queries.
- Chapter 6 Entity Framework/Core and LINQ to Entities
  - How to model database with object-relational mapping, and use functional LINQ queries to work with relational data in database.
  - How the C# LINQ to Entities queries are implemented to work with database.
  - How to change data in database, and handle concurrent conflicts.
  - Performance tips and asynchrony.
Part 3 Theories - demystifies the abstract mathematics theories, which are the rationale and foundations of LINQ and functional programming.
- Chapter 7 Lambda Calculus via C#
  - Core concepts of lambda calculus, bound and free variables, reduction (α-conversion, β-reduction, η-conversion), etc.
  - How to use lambda functions to represent values, data structures and computation, including Church Boolean, Church numbers, Church pair, Church list, and their operations.
  - Combinators and combinatory logic, including SKI combinator calculus, fixed point combinator for function recursion, etc.
- Chapter 8 Category Theory via C#
  - Core concepts of category theory, including category, object, morphism, monoid, functor, natural transformation, applicative functor, monad, and their laws.
  - How these concepts are applied in functional programming and LINQ.
  - How to manage I/O, state, exception handling, shared environment, logging, and continuation, etc., in functional programming.

This tutorial delivers highly reusable knowledge:

It covers C# language in depth, which can be generally applied in any programming paradigms besides functional programming.
It is a cross platform tutorial, covering both .NET Framework for Windows and .NET Core for Windows, Mac, Linux.
It demonstrates both usage and implementation of LINQ for mainstream data domains, which also enables developer to use the LINQ technologies for other data domains, or build custom LINQ APIs for specific data scenarios.
It also demystifies the abstract mathematics knowledge for functional programming, which applies to general functional programming, so it greatly helps developers understanding any other functional languages too.

As a fun of functional programming, LINQ, C#, and .NET technologies, hope this helps.

All code examples are available on GitHub: https://github.com/Dixin/CodeSnippets.

Functional programming and LINQ paradigm
1. Cross platform C# and .NET
  - Introducing cross platform .NET, C# and LINQ
    - .NET Framework, C#, and LINQ
    - .NET Core, UWP, Mono, Xamarin and Unity
    - .NET Standard
  - Introducing this book
    - Book structure
    - Code examples
  - Start coding
    - Start coding with Visual Studio (Windows)
    - Start coding with Visual Studio Code (Windows, macOS and Linux)
    - Start coding with Visual Studio for Mac (macOS)
2. Programming paradigms and functional programming
  - Programming paradigms
  - Imperative programming vs. declarative programming
  - Object-oriented programming vs. functional programming
3. LINQ to data sources
  - One language for different data domains
    - LINQ to Objects
    - Parallel LINQ
    - LINQ to XML
    - LINQ to DataSets
    - LINQ to Entities
    - LINQ to SQL
    - LINQ to NoSQL (LINQ to CosmosDB)
    - LINQ to JSON
    - LINQ to Twitter
  - Sequential query vs. parallel query
  - Local query vs. remote query
Functional programming in depth
1. C# language basics
 - Types and members
 - Types and members
 - Built-in types
 - Reference type vs. value type
 - ref local variable and immutable ref local variable
 - Array and stack-allocated array
 - Default value
 - ref structure
 - Static class
 - Partial type
 - Interface and implementation
 - IDisposable interface and using declaration
 - Generic type
 - Type parameter
 - Type parameter constraints
 - Nullable value type
 - Auto property
 - Property initializer
 - Object initializer
 - Collection initializer
 - Index initializer
 - Null coalescing operator
 - Null conditional operator
 - throw expression
 - Exception filter
 - String interpolation
 - nameof operator
 - Digit separator and leading underscore
2. Named function and function polymorphism
 - Constructor, static constructor and finalizer
 - Static method and instance method
 - Extension method
 - More named functions
 - Function polymorphisms
 - Ad hoc polymorphism: method overload
 - Parametric polymorphism: generic method
 - Type argument inference
 - Static import
 - Partial method
3. Local function and closure
 - Local function
 - Closure
 - Outer variable
 - Implicit reference
 - Static local function
4. Function input and output
 - Input by copy vs. input by alias (ref parameter)
 - Input by immutable alias (in parameter)
 - Output parameter (out parameter) and out variable
 - Discarding out variable
 - Parameter array
 - Positional argument vs. named argument
 - Required parameter vs. optional parameter
 - Caller information parameter
 - Output by copy vs. output by alias
 - Output by immutable alias
5. Delegate: Function type, instance, and group
 - Delegate type as function type
 - Function type
 - Generic delegate type
 - Unified built-in delegate types
 - Delegate instance as function instance
 - Delegate instance as function group
 - Event and event handler
6. Anonymous function and lambda expression
 - Anonymous method
 - Lambda expression as anonymous function
 - IIFE (Immediately-invoked function expression)
 - Closure
 - Expression bodied function member
7. Expression tree: Function as data
 - Lambda expression as expression tree
 - Metaprogramming: function as abstract syntax tree
 - .NET expressions
 - Compile expression tree at runtime
 - Traverse expression tree
 - Expression tree to CIL at runtime
 - Expression tree to executable function at runtime
 - Expression tree and LINQ remote query
8. Higher-order function, currying and first class function
 - First order function vs. higher-order function
 - Convert first-order function to higher-order function
 - Lambda operator => associativity
 - Curry function
 - Uncurry function
 - Partial applying function
 - First-class function
9. Function composition and chaining
 - Forward composition vs. backward composition
 - Forward piping
 - Method chaining and fluent interface
10. LINQ query expression
 - Syntax and compilation
 - Query expression pattern
 - LINQ query expression
 - Forward piping with LINQ
 - Query expression vs. query method
11. Covariance and contravariance
 - Subtyping and type polymorphism
 - Variances of non-generic function type
 - Variances of generic function type
 - Variances of generic interface
 - Variances of generic higher-order function type
 - Covariance of array
 - Variances in .NET and LINQ
12. Immutability, anonymous type and tuple
 - Immutable value
 - Constant local
 - Enumeration
 - using declaration and foreach statement
 - Immutable alias (immutable ref local variable)
 - Function’s immutable input and immutable output
 - Range variable in LINQ query expression
 - this reference for class
 - Immutable state (immutable type)
 - Constant field
 - Immutable class with readonly instance field
 - Immutable structure (readonly structure)
 - Immutable anonymous type
 - Local variable type inference
 - Immutable tuple vs. mutable tuple
 - Construction, element name and element inference
 - Deconstruction
 - Tuple assignment
 - Immutable collection vs. readonly collection
 - Shallow immutability vs. deep immutability
13. Pure function
 - Pure function vs. impure function
 - Referential transparency and side effect free
 - Purity in .NET
 - Purity in LINQ
14. Asynchronous function
 - Task, Task<TResult> and asynchrony
 - Named async function
 - Awaitable-awaiter pattern
 - Async state machine
 - Runtime context capture
 - Generalized async return type and async method builder
 - ValueTask<TResult> and performance
 - Anonymous async function
 - Asynchronous sequence: IAsyncEnumerable<T>
 - async using declaration: IAsyncDispose
15. Pattern matching
 - Is expression
 - switch statement and switch expression
LINQ to Objects: Querying objects in memory
1. Local sequential LINQ query
 - Iteration pattern and foreach statement
 - IEnumerable<T> and IEnumerator<T>
 - foreach loop vs. for loop
 - Non-generic sequence vs. generic sequence
 - LINQ to Objects queryable types
2. LINQ to Objects standard queries and query expressions
 - Sequence queries
 - Generation: Empty , Range, Repeat, DefaultIfEmpty
 - Filtering (restriction): Where, OfType, where
 - Mapping (projection): Select, SelectMany, from, let, select
 - Grouping: GroupBy, group, by, into
 - Join
 - Inner join: Join, SelectMany, join, on, equals
 - Outer join: GroupJoin, join, into, on, equals
 - Cross join: SelectMany, Join, from select, join, on, equals
 - Concatenation: Concat
 - Set: Distinct, Union, Intersect, Except
 - Convolution: Zip
 - Partitioning: Take, Skip, TakeWhile, SkipWhile
 - Ordering: OrderBy, ThenBy, OrderByDescending, ThenByDescending, Reverse, orderby, ascending, descending, into
 - Conversion: Cast, AsEnumerable
 - Collection queries
 - Conversion: ToArray, ToList, ToDictionary, ToLookup
 - Value queries
 - Element: First, FirstOrDefault, Last, LastOrDefault, ElementAt, ElementAtOrDefault, Single, SingleOrDefault
 - Aggregation: Aggregate, Count, LongCount, Min, Max, Sum, Average
 - Quantifier: All, Any, Contains
 - Equality: SequenceEqual
 - Queries in other languages
3. Generator
 - Implementing iterator pattern
 - Generating sequence and iterator
 - Yield statement and generator
4. Deferred execution, lazy evaluation and eager Evaluation
 - Immediate execution vs. Deferred execution
 - Cold IEnumerable<T> vs. hot IEnumerable<T>
 - Lazy evaluation vs. eager evaluation
5. LINQ to Objects internals: Standard queries implementation
 - Argument check and deferred execution
 - Collection queries
 - Conversion: ToArray, ToList, ToDictionary, ToLookup
 - Sequence queries
 - Conversion: Cast, AsEnumerable
 - Generation: Empty , Range, Repeat, DefaultIfEmpty
 - Filtering (restriction): Where, OfType
 - Mapping (projection): Select, SelectMany
 - Grouping: GroupBy
 - Join: SelectMany, Join, GroupJoin
 - Concatenation: Concat
 - Set: Distinct, Union, Intersect, Except
 - Convolution: Zip
 - Partitioning: Take, Skip, TakeWhile, SkipWhile
 - Ordering: OrderBy, ThenBy, OrderByDescending, ThenByDescending, Reverse
 - Value queries
 - Element: First, FirstOrDefault, Last, LastOrDefault, ElementAt, ElementAtOrDefault, Single, SingleOrDefault
 - Aggregation: Aggregate, Count, LongCount, Min, Max, Sum, Average
 - Quantifier: All, Any, Contains
 - Equality: SequenceEqual
6. Advanced queries in Microsoft Interactive Extensions (Ix)
 - Sequence queries
 - Generation: Defer, Create, Return, Repeat
 - Filtering: IgnoreElements, DistinctUntilChanged
 - Mapping: SelectMany, Scan, Expand
 - Concatenation: Concat, StartWith
 - Set: Distinct
 - Partitioning: TakeLast, SkipLast
 - Conversion: Hide
 - Buffering: Buffer, Share, Publish, Memoize
 - Exception: Throw, Catch, Finally, OnErrorResumeNext, Retry
 - Imperative: If, Case, Using, While, DoWhile, Generate, For
 - Iteration: Do
 - Value queries
 - Aggregation: Min, Max, MinBy, MaxBy
 - Quantifiers: isEmpty
 - Void queries
 - Iteration: ForEach
7. Building custom queries
 - Sequence queries (deferred execution)
 - Generation: Create, RandomInt32, RandomDouble, FromValue, FromValues, EmptyIfNull
 - Filtering: Timeout
 - Concatenation: Join, Append, Prepend, AppendTo, PrependTo
 - Partitioning: Subsequence
 - Exception: Catch, Retry
 - Comparison: OrderBy, OrderByDescending, ThenBy, ThenByDescending, GroupBy, Join, GroupJoin, Distinct, Union, Intersect, Except
 - List: Insert, Remove, RemoveAll, RemoveAt
 - Collection queries
 - Comparison: ToDictionary, ToLookup
 - Value queries
 - List: IndexOf, LastIndexOf
 - Aggregation: PercentileExclusive, PercentileInclusive, Percentile
 - Quantifiers: IsNullOrEmpty, IsNotNullOrEmpty
 - Comparison: Contains, SequenceEqual
 - Void queries
 - Iteration: ForEach
LINQ to XML: Querying XML
1. Modeling XML
  - Imperative vs. declarative paradigm
  - Types, conversions and operators
  - Read and deserialize XML
  - Serialize and write XML
  - Deferred construction
2. LINQ to XML standard queries
  - Navigation
  - Ordering
  - Comparison
  - More useful queries
  - XPath
    - Generate XPath expression
3. Manipulating XML
  - Clone
  - Adding, deleting, replacing, updating, and events
  - Annotation
  - Validating XML with XSD
  - Transforming XML with XSL
Parallel LINQ: Querying objects in parallel
1. Parallel LINQ query and visualization
  - Parallel query vs. sequential query
  - Parallel query execution
  - Visualizing parallel query execution
    - Using Concurrency Visualizer
    - Visualizing sequential and parallel LINQ queries
    - Visualizing chaining query methods
2. Parallel LINQ internals: data partitioning
  - Partitioning and load balancing
    - Range partitioning
    - Chunk partitioning
    - Hash partitioning
    - Stripped partitioning
  - Implement custom partitioner
    - Static partitioner
    - Dynamic partitioner
3. Parallel LINQ standard queries
  - Query settings
    - Cancellation
    - Degree of parallelism
    - Execution mode
    - Merge the values
  - Ordering
    - Preserving the order
    - Order and correctness
    - Orderable partitioner
  - Aggregation
    - Commutativity, associativity and correctness
    - Partitioning and merging
4. Parallel query performance
  - Sequential query vs. parallel query
  - CPU bound operation vs. IO bound operation
  - Factors to impact performance
Entity Framework/Core and LINQ to Entities: Querying relational data
1. Remote LINQ query
 - Entity Framework and Entity Framework Core
 - SQL database
 - Remote query vs. local query
 - Function vs. expression tree
2. Modeling database with object-relational mapping
 - Data types
 - Database
 - Connection resiliency and execution retry strategy
 - Tables
 - Relationships
 - One-to-one
 - One-to-many
 - Many-to-many
 - Inheritance
 - Views
3. Logging and tracing LINQ to Entities queries
 - Application side logging
 - Database side tracing with Extended Events
4. LINQ to Entities standard queries
 - Sequence queries
 - Generation: DefaultIfEmpty
 - Filtering (restriction): Where, OfType
 - Mapping (projection): Select
 - Grouping: GroupBy
 - Join
 - Inner join: Join, SelectMany, GroupJoin, Select
 - Outer join: GroupJoin, Select, SelectMany
 - Cross join and self join: SelectMany, Join
 - Concatenation: Concat
 - Set: Distinct, Union, Intersect, Except
 - Partitioning: Take, Skip
 - Ordering: OrderBy, ThenBy, OrderByDescending, ThenByDescending
 - Conversion: Cast, AsQueryable
 - Value queries
 - Element: First, FirstOrDefault, Single, SingleOrDefault
 - Aggregation: Count, LongCount, Min, Max, Sum, Average
 - Quantifier: All, Any, Contains
5. LINQ to Entities internals: Query translation implementation
 - Code to LINQ expression tree
 - IQueryable<T> and IQueryProvider
 - Standard remote queries
 - Building LINQ to Entities abstract syntax tree
 - .NET expression tree to database expression tree
 - Database query abstract syntax tree
 - Compiling LINQ expressions to database expressions
 - Compiling LINQ queries
 - Compiling .NET API calls
 - Remote API call vs. local API call
 - Compile database functions and operators
 - Database expression tree to database query language
 - SQL generator and SQL command
 - Generating SQL from database expression tree
6. Loading query data
 - Deferred execution
 - Iterator pattern
 - Lazy evaluation vs. eager evaluation
 - Explicit loading
 - Eager loading
 - Lazy loading
 - The N + 1 problem
 - Disabling lazy loading
7. Manipulating relational data: Data change and transaction
 - Repository pattern and unit of work pattern
 - Tracking entities and changes
 - Tracking entities
 - Tracking entity changes and property changes
 - Tracking relationship changes
 - Enabling and disabling tracking
 - Change data
 - Create
 - Update
 - Delete
 - Transaction
 - Transaction with connection resiliency and execution strategy
 - EF Core transaction
 - ADO.NET transaction
 - Transaction scope
8. Resolving optimistic concurrency
 - Detecting concurrent conflicts
 - Resolving concurrent conflicts
 - Retaining database values (database wins)
 - Overwriting database values (client wins)
 - Merging with database values
 - Saving changes with concurrent conflict handling
Lambda Calculus via C#: The foundation of all functional programming
1. Basics
  - Expression
    - Bound variable vs. free variable
  - Reductions
    - α-conversion (alpha-conversion)
    - β-reduction (beta-reduction)
    - η-conversion (eta-conversion)
    - Normal order
    - Applicative order
  - Function composition
    - Associativity
    - Unit
2. Church encoding: Function as boolean and logic
  - Church encoding
  - Church Boolean
  - Logical operators
  - Conversion between Church Boolean and System.Boolean
  - If
3. Church encoding: Function as numeral, arithmetic and predicate
  - Church numerals
  - Increase and decrease
  - Arithmetic operators
  - Predicate and relational operators
    - Attempt of recursion
  - Conversion between Church numeral and System.UInt32
4. Church encoding: Function as tuple and signed numeral
  - Church pair (2-tuple)
    - Tuple operators
  - N-tuple
  - Signed numeral
    - Arithmetic operators
5. Church encoding: Function as list
  - Tuple as list node
    - List operators
  - Aggregation function as list node
    - List operators
  - Model everything
6. Combinatory logic
  - Combinator
  - SKI combinator calculus
    - SKI compiler: compile lambda calculus expression to SKI calculus combinator
  - Iota combinator calculus
7. Fixed point combinator and recursion
  - Normal order fixed point combinator (Y combinator) and recursion
  - Applicative order fixed point combinator (Z combinator) and recursion
8. Undecidability of equivalence
  - Halting problem
  - Equivalence problem
Category Theory via C#: The essentials and design of LINQ
1. Basics: Category and morphism
 - Category and category laws
 - DotNet category
2. Monoid
 - Monoid and monoid laws
 - Monoid as category
3. Functor and LINQ to Functors
 - Functor and functor laws
 - Endofunctor
 - Type constructor and higher-kinded type
 - LINQ to Functors
 - Built-in IEnumerable<> functor
 - Functor pattern of LINQ
 - More LINQ to Functors
4. Natural transformation
 - Natural transformation and naturality
 - Functor Category
 - Endofunctor category
5. Bifunctor
 - Bifunctor
 - Monoidal category
6. Monoidal functor and applicative functor
 - Monoidal functor
 - IEnumeable<> monoidal functor
 - Applicative functor
 - IEnumeable<> applicative functor
 - Monoidal functor vs. applicative functor
 - More Monoidal functors and applicative functors
7. Monad and LINQ to Monads
 - Monad
 - LINQ to Monads and monad laws
 - Built-in IEnumerable<> monad
 - Monad laws and Kleisli composition
 - Kleisli category
 - Monad pattern of LINQ
 - Monad vs. monoidal/applicative functor
 - More LINQ to Monads
8. Advanced LINQ to Monads
 - IO monad
 - State monad
 - Try monad
 - Reader monad
 - Writer monad
 - Continuation monad

Understanding (all) JavaScript module formats and tools

Sun, 09 Feb 2025 00:00:00 GMT

When you build an application with JavaScript, you always want to modularize your code. However, JavaScript language was initially invented for simple form manipulation, with no built-in features like module or namespace. In years, tons of technologies are invented to modularize JavaScript. This article discusses all mainstream terms, patterns, libraries, syntax, and tools for JavaScript modules.

IIFE module: JavaScript module pattern
- IIFE: Immediately invoked function expression
- Import mixins
Revealing module: JavaScript revealing module pattern
CJS module: CommonJS module, or Node.js module
AMD module: Asynchronous Module Definition, or RequireJS module
- Dynamic loading
- AMD module from CommonJS module
UMD module: Universal Module Definition, or UmdJS module
- UMD for both AMD (RequireJS) and native browser
- UMD for both AMD (RequireJS) and CommonJS (Node.js)
ES module: ECMAScript 2015, or ES6 module
ES dynamic module: ECMAScript 2020, or ES11 dynamic module
System module: SystemJS module
- Dynamic module loading
Webpack module: bundle from CJS, AMD, ES modules
Babel module: transpile from ES module
- Babel with SystemJS
TypeScript module: Transpile to CJS, AMD, ES, System modules
- Internal module and namespace
Conclusion

IIFE module: JavaScript module pattern

In the browser, defining a JavaScript variable is defining a global variable, which causes pollution across all JavaScript files loaded by the current web page:

// Define global variables.
let count = 0;
const increase = () => ++count;
const reset = () => {
    count = 0;
    console.log("Count is reset.");
};

// Use global variables.
increase();
reset();

To avoid global pollution, an anonymous function can be used to wrap the code:

(() => {
    let count = 0;
    // ...
});

Apparently, there is no longer any global variable. However, defining a function does not execute the code inside the function.

IIFE: Immediately invoked function expression

To execute the code inside a function f, the syntax is function call () as f(). To execute the code inside an anonymous function (() => {}), the same function call syntax () can be used as (() => {})():

(() => {
    let count = 0;
    // ...
})();

This is called an IIFE (Immediately invoked function expression). So a basic module can be defined in this way:

// Define IIFE module.
const iifeCounterModule = (() => {
    let count = 0;
    return {
        increase: () => ++count,
        reset: () => {
            count = 0;
            console.log("Count is reset.");
        }
    };
})();

// Use IIFE module.
iifeCounterModule.increase();
iifeCounterModule.reset();

It wraps the module code inside an IIFE. The anonymous function returns an object, which is the placeholder of exported APIs. Only 1 global variable is introduced, which is the module name (or namespace). Later the module name can be used to call the exported module APIs. This is called the module pattern of JavaScript.

Import mixins

When defining a module, some dependencies may be required. With IIFE module pattern, each dependent module is a global variable. The dependent modules can be directly accessed inside the anonymous function, or they can be passed as the anonymous function’s arguments:

// Define IIFE module with dependencies.
const iifeCounterModule = ((dependencyModule1, dependencyModule2) => {
    let count = 0;
    return {
        increase: () => ++count,
        reset: () => {
            count = 0;
            console.log("Count is reset.");
        }
    };
})(dependencyModule1, dependencyModule2);

The early version of popular libraries, like jQuery, followed this pattern. (The latest version of jQuery follows the UMD module, which is explained later in this article.)

Revealing module: JavaScript revealing module pattern

The revealing module pattern is named by Christian Heilmann. This pattern is also an IIFE, but it emphasizes defining all APIs as local variables inside the anonymous function:

// Define revealing module.
const revealingCounterModule = (() => {
    let count = 0;
    const increase = () => ++count;
    const reset = () => {
        count = 0;
        console.log("Count is reset.");
    };

    return {
        increase,
        reset
    };
})();

// Use revealing module.
revealingCounterModule.increase();
revealingCounterModule.reset();

With this syntax, it becomes easier when the APIs need to call each other.

CJS module: CommonJS module, or Node.js module

CommonJS, initially named ServerJS, is a pattern to define and consume modules. It is implemented by Node,js. By default, each .js file is a CommonJS module. A module variable and an exports variable are provided for a module (a file) to expose APIs. And a require function is provided to load and consume a module. The following code defines the counter module in CommonJS syntax:

// Define CommonJS module: commonJSCounterModule.js.
const dependencyModule1 = require("./dependencyModule1");
const dependencyModule2 = require("./dependencyModule2");

let count = 0;
const increase = () => ++count;
const reset = () => {
    count = 0;
    console.log("Count is reset.");
};

exports.increase = increase;
exports.reset = reset;
// Or equivalently:
module.exports = {
    increase,
    reset
};

The following example consumes the counter module:

// Use CommonJS module.
const { increase, reset } = require("./commonJSCounterModule");
increase();
reset();
// Or equivelently:
const commonJSCounterModule = require("./commonJSCounterModule");
commonJSCounterModule.increase();
commonJSCounterModule.reset();

At runtime, Node.js implements this by wrapping the code inside the file into a function, then passes the exports variable, module variable, and require function through arguments.

// Define CommonJS module: wrapped commonJSCounterModule.js.
(function (exports, require, module, __filename, __dirname) {
    const dependencyModule1 = require("./dependencyModule1");
    const dependencyModule2 = require("./dependencyModule2");

    let count = 0;
    const increase = () => ++count;
    const reset = () => {
        count = 0;
        console.log("Count is reset.");
    };

    module.exports = {
        increase,
        reset
    };

    return module.exports;
}).call(thisValue, exports, require, module, filename, dirname);

// Use CommonJS module.
(function (exports, require, module, __filename, __dirname) {
    const commonJSCounterModule = require("./commonJSCounterModule");
    commonJSCounterModule.increase();
    commonJSCounterModule.reset();
}).call(thisValue, exports, require, module, filename, dirname);

AMD module: Asynchronous Module Definition, or RequireJS module

AMD (Asynchronous Module Definition https://github.com/amdjs/amdjs-api), is a pattern to define and consume module. It is implemented by RequireJS library https://requirejs.org/. AMD provides a define function to define module, which accepts the module name, dependent modules’ names, and a factory function:

// Define AMD module.
define("amdCounterModule", ["dependencyModule1", "dependencyModule2"], (dependencyModule1, dependencyModule2) => {
    let count = 0;
    const increase = () => ++count;
    const reset = () => {
        count = 0;
        console.log("Count is reset.");
    };

    return {
        increase,
        reset
    };
});

It also provides a require function to consume module:

// Use AMD module.
require(["amdCounterModule"], amdCounterModule => {
    amdCounterModule.increase();
    amdCounterModule.reset();
});

The AMD require function is totally different from the CommonJS require function. AMD require accept the names of modules to be consumed, and pass the module to a function argument.

Dynamic loading

AMD’s define function has another overload. It accepts a callback function, and pass a CommonJS-like require function to that callback. Inside the callback function, require can be called to dynamically load the module:

// Use dynamic AMD module.
define(require => {
    const dynamicDependencyModule1 = require("dependencyModule1");
    const dynamicDependencyModule2 = require("dependencyModule2");

    let count = 0;
    const increase = () => ++count;
    const reset = () => {
        count = 0;
        console.log("Count is reset.");
    };

    return {
        increase,
        reset
    };
});

AMD module from CommonJS module

The above define function overload can also passes the require function as well as exports variable and module to its callback function. So inside the callback, CommonJS syntax code can work:

// Define AMD module with CommonJS code.
define((require, exports, module) => {
    // CommonJS code.
    const dependencyModule1 = require("dependencyModule1");
    const dependencyModule2 = require("dependencyModule2");

    let count = 0;
    const increase = () => ++count;
    const reset = () => {
        count = 0;
        console.log("Count is reset.");
    };

    exports.increase = increase;
    exports.reset = reset;
});

// Use AMD module with CommonJS code.
define(require => {
    // CommonJS code.
    const counterModule = require("amdCounterModule");
    counterModule.increase();
    counterModule.reset();
});

UMD module: Universal Module Definition, or UmdJS module

UMD (Universal Module Definition, https://github.com/umdjs/umd) is a set of tricky patterns to make your code file work in multiple environments.

UMD for both AMD (RequireJS) and native browser

For example, the following is a kind of UMD pattern to make module definition work with both AMD (RequireJS) and native browser:

// Define UMD module for both AMD and browser.
((root, factory) => {
    // Detects AMD/RequireJS"s define function.
    if (typeof define === "function" && define.amd) {
        // Is AMD/RequireJS. Call factory with AMD/RequireJS"s define function.
        define("umdCounterModule", ["deependencyModule1", "dependencyModule2"], factory);
    } else {
        // Is Browser. Directly call factory.
        // Imported dependencies are global variables(properties of window object).
        // Exported module is also a global variable(property of window object)
        root.umdCounterModule = factory(root.deependencyModule1, root.dependencyModule2);
    }
})(typeof self !== "undefined" ? self : this, (deependencyModule1, dependencyModule2) => {
    // Module code goes here.
    let count = 0;
    const increase = () => ++count;
    const reset = () => {
        count = 0;
        console.log("Count is reset.");
    };

    return {
        increase,
        reset
    };
});

It is more complex but it is just an IIFE. The anonymous function detects if AMD’s define function exists.

If yes, call the module factory with AMD’s define function.
If not, it calls the module factory directly. At this moment, the root argument is actually the browser’s window object. It gets dependency modules from global variables (properties of window object). When factory returns the module, the returned module is also assigned to a global variable (property of window object).

UMD for both AMD (RequireJS) and CommonJS (Node.js)

The following is another kind of UMD pattern to make module definition work with both AMD (RequireJS) and CommonJS (Node.js):

(define => define((require, exports, module) => {
    // Module code goes here.
    const dependencyModule1 = require("dependencyModule1");
    const dependencyModule2 = require("dependencyModule2");

    let count = 0;
    const increase = () => ++count;
    const reset = () => {
        count = 0;
        console.log("Count is reset.");
    };

    module.export = {
        increase,
        reset
    };
}))(// Detects module variable and exports variable of CommonJS/Node.js.
    // Also detect the define function of AMD/RequireJS.
    typeof module === "object" && module.exports && typeof define !== "function"
        ? // Is CommonJS/Node.js. Manually create a define function.
            factory => module.exports = factory(require, exports, module)
        : // Is AMD/RequireJS. Directly use its define function.
            define);

Again, don’t be scared. It is just another IIFE. When the anonymous function is called, its argument is evaluated. The argument evaluation detects the environment (check the module variable and exports variable of CommonJS/Node.js, as well as the define function of AMD/RequireJS).

If the environment is CommonJS/Node.js, the anonymous function’s argument is a manually created define function.
If the environment is AMD/RequireJS, the anonymous function’s argument is just AMD’s define function. So when the anonymous function is executed, it is guaranteed to have a working define function. Inside the anonymous function, it simply calls the define function to create the module.

ES module: ECMAScript 2015, or ES6 module

After all the module mess, in 2015, JavaScript’s spec version 6 introduces one more different module syntax. This spec is called ECMAScript 2015 (ES2015), or ECMAScript 6 (ES6). The main syntax is the import keyword and the export keyword. The following example uses new syntax to demonstrate ES module’s named import/export and default import/export:

// Define ES module: esCounterModule.js or esCounterModule.mjs.
import dependencyModule1 from "./dependencyModule1.mjs";
import dependencyModule2 from "./dependencyModule2.mjs";

let count = 0;
// Named export:
export const increase = () => ++count;
export const reset = () => {
    count = 0;
    console.log("Count is reset.");
};
// Or default export:
export default {
    increase,
    reset
};

To use this module file in browser, add a <script> tag and specify it is a module: <script type="module" src="esCounterModule.js"></script>. To use this module file in Node.js, rename its extension from .js to .mjs.

// Use ES module.
// Browser: <script type="module" src="esCounterModule.js"></script> or inline.
// Server: esCounterModule.mjs
// Import from named export.
import { increase, reset } from "./esCounterModule.mjs";
increase();
reset();
// Or import from default export:
import esCounterModule from "./esCounterModule.mjs";
esCounterModule.increase();
esCounterModule.reset();

For browser, <script>’s nomodule attribute can be used for fallback:

<script nomodule>
    alert("Not supported.");
</script>

ES dynamic module: ECMAScript 2020, or ES11 dynamic module

In 2020, the latest JavaScript spec version 11 is introducing a built-in function import to consume an ES module dynamically. The import function returns a promise, so its then method can be called to consume the module:

// Use dynamic ES module with promise APIs, import from named export:
import("./esCounterModule.js").then(({ increase, reset }) => {
    increase();
    reset();
});
// Or import from default export:
import("./esCounterModule.js").then(dynamicESCounterModule => {
    dynamicESCounterModule.increase();
    dynamicESCounterModule.reset();
});

By returning a promise, apparently, import function can also work with the await keyword:

// Use dynamic ES module with async/await.
(async () => {

    // Import from named export:
    const { increase, reset } = await import("./esCounterModule.js");
    increase();
    reset();

    // Or import from default export:
    const dynamicESCounterModule = await import("./esCounterModule.js");
    dynamicESCounterModule.increase();
    dynamicESCounterModule.reset();

})();

The following is the compatibility of import/dynamic import/export, from https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules:

System module: SystemJS module

SystemJS is a library that can enable ES module syntax for older ES. For example, the following module is defined in ES 6 syntax:

// Define ES module.
import dependencyModule1 from "./dependencyModule1.js";
import dependencyModule2 from "./dependencyModule2.js";
dependencyModule1.api1();
dependencyModule2.api2();

let count = 0;
// Named export:
export const increase = function () { return ++count };
export const reset = function () {
    count = 0;
    console.log("Count is reset.");
};
// Or default export:
export default {
    increase,
    reset
}

If the current runtime, like an old browser, does not support ES6 syntax, the above code cannot work. One solution is to transpile the above module definition to a call of SystemJS library API, System.register:

// Define SystemJS module.
System.register(["./dependencyModule1.js", "./dependencyModule2.js"], function (exports_1, context_1) {
    "use strict";
    var dependencyModule1_js_1, dependencyModule2_js_1, count, increase, reset;
    var __moduleName = context_1 && context_1.id;
    return {
        setters: [
            function (dependencyModule1_js_1_1) {
                dependencyModule1_js_1 = dependencyModule1_js_1_1;
            },
            function (dependencyModule2_js_1_1) {
                dependencyModule2_js_1 = dependencyModule2_js_1_1;
            }
        ],
        execute: function () {
            dependencyModule1_js_1.default.api1();
            dependencyModule2_js_1.default.api2();
            count = 0;
            // Named export:
            exports_1("increase", increase = function () { return ++count };
            exports_1("reset", reset = function () {
                count = 0;
                console.log("Count is reset.");
            };);
            // Or default export:
            exports_1("default", {
                increase,
                reset
            });
        }
    };
});

So that the import/export new ES6 syntax is gone. The old API call syntax works for sure. This transpilation can be done automatically with Webpack, TypeScript, etc., which are explained later in this article.

Dynamic module loading

SystemJS also provides an import function for dynamic import:

// Use SystemJS module with promise APIs.
System.import("./esCounterModule.js").then(dynamicESCounterModule => {
    dynamicESCounterModule.increase();
    dynamicESCounterModule.reset();
});

Webpack module: bundle from CJS, AMD, ES modules

Webpack is a bundler for modules. It transpiles combined CommonJS module, AMD module, and ES module into a single harmony module pattern, and bundle all code into a single file. For example, the following 3 files define 3 modules in 3 different syntaxes:

// Define AMD module: amdDependencyModule1.js
define("amdDependencyModule1", () => {
    const api1 = () => { };
    return {
        api1
    };
});

// Define CommonJS module: commonJSDependencyModule2.js
const dependencyModule1 = require("./amdDependencyModule1");
const api2 = () => dependencyModule1.api1();
exports.api2 = api2;

// Define ES module: esCounterModule.js.
import dependencyModule1 from "./amdDependencyModule1";
import dependencyModule2 from "./commonJSDependencyModule2";
dependencyModule1.api1();
dependencyModule2.api2();

let count = 0;
const increase = () => ++count;
const reset = () => {
    count = 0;
    console.log("Count is reset.");
};

export default {
    increase,
    reset
}

And the following file consumes the counter module:

// Use ES module: index.js
import counterModule from "./esCounterModule";
counterModule.increase();
counterModule.reset();

Webpack can bundle all the above file, even they are in 3 different module systems, into a single file main.js:

root
- dist
  - main.js (Bundle of all files under src)
- src
  - amdDependencyModule1.js
  - commonJSDependencyModule2.js
  - esCounterModule.js
  - index.js
- webpack.config.js

Since Webpack is based on Node.js, Webpack uses CommonJS module syntax for itself. In webpack.config.js:

const path = require('path');

module.exports = {
    entry: './src/index.js',
    mode: "none", // Do not optimize or minimize the code for readability.
    output: {
        filename: 'main.js',
        path: path.resolve(__dirname, 'dist'),
    },
};

Now run the following command to transpile and bundle all 4 files, which are in different syntax:

npm install webpack webpack-cli --save-dev
npx webpack --config webpack.config.js

AS a result, Webpack generates the bundle file main.js. The following code in main.js is reformatted, and variables are renamed, to improve readability:

(function (modules) { // webpackBootstrap
    // The module cache
    var installedModules = {};
    // The require function
    function require(moduleId) {
        // Check if module is in cache
        if (installedModules[moduleId]) {
            return installedModules[moduleId].exports;

        }
        // Create a new module (and put it into the cache)
        var module = installedModules[moduleId] = {
            i: moduleId,
            l: false,
            exports: {}

        };
        // Execute the module function
        modules[moduleId].call(module.exports, module, module.exports, require);
        // Flag the module as loaded
        module.l = true;
        // Return the exports of the module
        return module.exports;
    }

    // expose the modules object (__webpack_modules__)
    require.m = modules;
    // expose the module cache
    require.c = installedModules;
    // define getter function for harmony exports
    require.d = function (exports, name, getter) {
        if (!require.o(exports, name)) {
            Object.defineProperty(exports, name, { enumerable: true, get: getter });

        }

    };
    // define __esModule on exports
    require.r = function (exports) {
        if (typeof Symbol !== 'undefined' && Symbol.toStringTag) {
            Object.defineProperty(exports, Symbol.toStringTag, { value: 'Module' });

        }
        Object.defineProperty(exports, '__esModule', { value: true });

    };
    // create a fake namespace object
    // mode & 1: value is a module id, require it
    // mode & 2: merge all properties of value into the ns
    // mode & 4: return value when already ns object
    // mode & 8|1: behave like require
    require.t = function (value, mode) {
        if (mode & 1) value = require(value);
        if (mode & 8) return value;
        if ((mode & 4) && typeof value === 'object' && value && value.__esModule) return value;
        var ns = Object.create(null);
        require.r(ns);
        Object.defineProperty(ns, 'default', { enumerable: true, value: value });
        if (mode & 2 && typeof value != 'string') for (var key in value) require.d(ns, key, function (key) { return value[key]; }.bind(null, key));
        return ns;
    };
    // getDefaultExport function for compatibility with non-harmony modules
    require.n = function (module) {
        var getter = module && module.__esModule ?
            function getDefault() { return module['default']; } :
            function getModuleExports() { return module; };
        require.d(getter, 'a', getter);
        return getter;
    };
    // Object.prototype.hasOwnProperty.call
    require.o = function (object, property) { return Object.prototype.hasOwnProperty.call(object, property); };
    // __webpack_public_path__
    require.p = "";
    // Load entry module and return exports
    return require(require.s = 0);
})([
    function (module, exports, require) {
        "use strict";
        require.r(exports);
        // Use ES module: index.js.
        var esCounterModule = require(1);
        esCounterModule["default"].increase();
        esCounterModule["default"].reset();
    },
    function (module, exports, require) {
        "use strict";
        require.r(exports);
        // Define ES module: esCounterModule.js.
        var amdDependencyModule1 = require.n(require(2));
        var commonJSDependencyModule2 = require.n(require(3));
        amdDependencyModule1.a.api1();
        commonJSDependencyModule2.a.api2();

        let count = 0;
        const increase = () => ++count;
        const reset = () => {
            count = 0;
            console.log("Count is reset.");
        };

        exports["default"] = {
            increase,
            reset
        };
    },
    function (module, exports, require) {
        var result;
        !(result = (() => {
            // Define AMD module: amdDependencyModule1.js
            const api1 = () => { };
            return {
                api1
            };
        }).call(exports, require, exports, module),
            result !== undefined && (module.exports = result));
    },
    function (module, exports, require) {
        // Define CommonJS module: commonJSDependencyModule2.js
        const dependencyModule1 = require(2);
        const api2 = () => dependencyModule1.api1();
        exports.api2 = api2;
    }
]);

Again, it is just another IIFE. The code of all 4 files is transpiled to the code in 4 functions in an array. And that array is passed to the anonymous function as an argument.

Babel module: transpile from ES module

Babel is another transpiler to convert ES6+ JavaScript code to the older syntax for the older environment like older browsers. The above counter module in ES6 import/export syntax can be converted to the following babel module with new syntax replaced:

// Babel.
Object.defineProperty(exports, "__esModule", {
    value: true
});
exports["default"] = void 0;
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }

// Define ES module: esCounterModule.js.
var dependencyModule1 = _interopRequireDefault(require("./amdDependencyModule1"));
var dependencyModule2 = _interopRequireDefault(require("./commonJSDependencyModule2"));
dependencyModule1["default"].api1();
dependencyModule2["default"].api2();

var count = 0;
var increase = function () { return ++count; };
var reset = function () {
    count = 0;
    console.log("Count is reset.");
};

exports["default"] = {
    increase: increase,
    reset: reset
};

And here is the code in index.js which consumes the counter module:

// Babel.
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }

// Use ES module: index.js
var esCounterModule = _interopRequireDefault(require("./esCounterModule.js"));
esCounterModule["default"].increase();
esCounterModule["default"].reset();

This is the default transpilation. Babel can also work with other tools.

Babel with SystemJS

SystemJS can be used as a plugin for Babel:

npm install --save-dev @babel/plugin-transform-modules-systemjs

And it should be added to the Babel configuration babel.config.json:

{
    "plugins": ["@babel/plugin-transform-modules-systemjs"],
    "presets": [
        [
            "@babel/env",
            {
                "targets": {
                    "ie": "11"
                }
            }
        ]
    ]
}

Now Babel can work with SystemJS to transpile CommonJS/Node.js module, AMD/RequireJS module, and ES module:

npx babel src --out-dir lib

The result is:

root
- lib
  - amdDependencyModule1.js (Transpiled with SystemJS)
  - commonJSDependencyModule2.js (Transpiled with SystemJS)
  - esCounterModule.js (Transpiled with SystemJS)
  - index.js (Transpiled with SystemJS)
- src
  - amdDependencyModule1.js
  - commonJSDependencyModule2.js
  - esCounterModule.js
  - index.js
- babel.config.json

Now all the ADM, CommonJS, and ES module syntax are transpiled to SystemJS syntax:

// Transpile AMD/RequireJS module definition to SystemJS syntax: lib/amdDependencyModule1.js.
System.register([], function (_export, _context) {
    "use strict";
    return {
        setters: [],
        execute: function () {
            // Define AMD module: src/amdDependencyModule1.js
            define("amdDependencyModule1", () => {
                const api1 = () => { };

                return {
                    api1
                };
            });
        }
    };
});

// Transpile CommonJS/Node.js module definition to SystemJS syntax: lib/commonJSDependencyModule2.js.
System.register([], function (_export, _context) {
    "use strict";
    var dependencyModule1, api2;
    return {
        setters: [],
        execute: function () {
            // Define CommonJS module: src/commonJSDependencyModule2.js
            dependencyModule1 = require("./amdDependencyModule1");

            api2 = () => dependencyModule1.api1();

            exports.api2 = api2;
        }
    };
});

// Transpile ES module definition to SystemJS syntax: lib/esCounterModule.js.
System.register(["./amdDependencyModule1", "./commonJSDependencyModule2"], function (_export, _context) {
    "use strict";
    var dependencyModule1, dependencyModule2, count, increase, reset;
    return {
        setters: [function (_amdDependencyModule) {
            dependencyModule1 = _amdDependencyModule.default;
        }, function (_commonJSDependencyModule) {
            dependencyModule2 = _commonJSDependencyModule.default;
        }],
        execute: function () {
            // Define ES module: src/esCounterModule.js.
            dependencyModule1.api1();
            dependencyModule2.api2();
            count = 0;

            increase = () => ++count;

            reset = () => {
                count = 0;
                console.log("Count is reset.");
            };

            _export("default", {
                increase,
                reset
            });
        }
    };
});

// Transpile ES module usage to SystemJS syntax: lib/index.js.
System.register(["./esCounterModule"], function (_export, _context) {
    "use strict";
    var esCounterModule;
    return {
        setters: [function (_esCounterModuleJs) {
            esCounterModule = _esCounterModuleJs.default;
        }],
        execute: function () {
            // Use ES module: src/index.js
            esCounterModule.increase();
            esCounterModule.reset();
        }
    };
});

TypeScript module: Transpile to CJS, AMD, ES, System modules

TypeScript supports all JavaScript syntax, including the ES6 module syntax https://www.typescriptlang.org/docs/handbook/modules.html. When TypeScript transpiles, the ES module code can either be kept as ES6, or transpiled to other formats, including CommonJS/Node.js, AMD/RequireJS, UMD/UmdJS, or System/SystemJS, according to the specified transpiler options in tsconfig.json:

{
    "compilerOptions": {
        "module": "ES2020", // None, CommonJS, AMD, System, UMD, ES6, ES2015, ES2020, ESNext.
    }
}

For example:

// TypeScript and ES module.
// With compilerOptions: { module: "ES6" }. Transpile to ES module with the same import/export syntax.
import dependencyModule from "./dependencyModule";
dependencyModule.api();
let count = 0;
export const increase = function () { return ++count };


// With compilerOptions: { module: "CommonJS" }. Transpile to CommonJS/Node.js module:
var __importDefault = (this && this.__importDefault) || function (mod) {
    return (mod && mod.__esModule) ? mod : { "default": mod };
};
exports.__esModule = true;

var dependencyModule_1 = __importDefault(require("./dependencyModule"));
dependencyModule_1["default"].api();
var count = 0;
exports.increase = function () { return ++count; };

// With compilerOptions: { module: "AMD" }. Transpile to AMD/RequireJS module:
var __importDefault = (this && this.__importDefault) || function (mod) {
    return (mod && mod.__esModule) ? mod : { "default": mod };
};
define(["require", "exports", "./dependencyModule"], function (require, exports, dependencyModule_1) {
    "use strict";
    exports.__esModule = true;

    dependencyModule_1 = __importDefault(dependencyModule_1);
    dependencyModule_1["default"].api();
    var count = 0;
    exports.increase = function () { return ++count; };
});

// With compilerOptions: { module: "UMD" }. Transpile to UMD/UmdJS module:
var __importDefault = (this && this.__importDefault) || function (mod) {
    return (mod && mod.__esModule) ? mod : { "default": mod };
};
(function (factory) {
    if (typeof module === "object" && typeof module.exports === "object") {
        var v = factory(require, exports);
        if (v !== undefined) module.exports = v;
    }
    else if (typeof define === "function" && define.amd) {
        define(["require", "exports", "./dependencyModule"], factory);
    }
})(function (require, exports) {
    "use strict";
    exports.__esModule = true;

    var dependencyModule_1 = __importDefault(require("./dependencyModule"));
    dependencyModule_1["default"].api();
    var count = 0;
    exports.increase = function () { return ++count; };
});

// With compilerOptions: { module: "System" }. Transpile to System/SystemJS module:
System.register(["./dependencyModule"], function (exports_1, context_1) {
    "use strict";
    var dependencyModule_1, count, increase;
    var __moduleName = context_1 && context_1.id;
    return {
        setters: [
            function (dependencyModule_1_1) {
                dependencyModule_1 = dependencyModule_1_1;
            }
        ],
        execute: function () {
            dependencyModule_1["default"].api();
            count = 0;
            exports_1("increase", increase = function () { return ++count; });
        }
    };
});

The ES module syntax supported in TypeScript was called external modules.

Internal module and namespace

TypeScript also has a module keyword and a namespace keyword https://www.typescriptlang.org/docs/handbook/namespaces-and-modules.html#pitfalls-of-namespaces-and-modules. They were called internal modules:

module Counter {
    let count = 0;
    export const increase = () => ++count;
    export const reset = () => {
        count = 0;
        console.log("Count is reset.");
    };
}

namespace Counter {
    let count = 0;
    export const increase = () => ++count;
    export const reset = () => {
        count = 0;
        console.log("Count is reset.");
    };
}

They are both transpiled to JavaScript objects:

var Counter;
(function (Counter) {
    var count = 0;
    Counter.increase = function () { return ++count; };
    Counter.reset = function () {
        count = 0;
        console.log("Count is reset.");
    };
})(Counter || (Counter = {}));

TypeScript module and namespace can have multiple levels by supporting the . separator:

module Counter.Sub {
    let count = 0;
    export const increase = () => ++count;
}

namespace Counter.Sub {
    let count = 0;
    export const increase = () => ++count;
}

The the sub module and sub namespace are both transpiled to object’s property:

var Counter;
(function (Counter) {
    var Sub;
    (function (Sub) {
        var count = 0;
        Sub.increase = function () { return ++count; };
    })(Sub = Counter.Sub || (Counter.Sub = {}));
})(Counter|| (Counter = {}));

TypeScript module and namespace can also be used in the export statement:

module Counter {
    let count = 0;
    export module Sub {
        export const increase = () => ++count;
    }
}

module Counter {
    let count = 0;
    export namespace Sub {
        export const increase = () => ++count;
    }
}

The transpilation is the same as submodule and sub-namespace:

var Counter;
(function (Counter) {
    var count = 0;
    var Sub;
    (function (Sub) {
        Sub.increase = function () { return ++count; };
    })(Sub = Counter.Sub || (Counter.Sub = {}));
})(Counter || (Counter = {}));

Conclusion

Welcome to JavaScript, which has so much drama - 10+ systems/formats just for modularization/namespace:

IIFE module: JavaScript module pattern
Revealing module: JavaScript revealing module pattern
CJS module: CommonJS module, or Node.js module
AMD module: Asynchronous Module Definition, or RequireJS module
UMD module: Universal Module Definition, or UmdJS module
ES module: ECMAScript 2015, or ES6 module
ES dynamic module: ECMAScript 2020, or ES11 dynamic module
System module: SystemJS module
Webpack module: transpile and bundle of CJS, AMD, ES modules
Babel module: transpile ES module
TypeScript module and namespace

Fortunately, now JavaScript has standard built-in language features for modules, and it is supported by Node.js and all the latest modern browsers. For the older environments, you can still code with the new ES module syntax, then use Webpack/Babel/SystemJS/TypeScript to transpile to older or compatible syntax.

Port Microsoft Concurrency Visualizer SDK to .NET Standard and NuGet

Fri, 24 Jan 2025 00:00:00 GMT

I uploaded a NuGet package of Microsoft Concurrency Visualizer SDK: ConcurrencyVisualizer. Microsoft Concurrency Visualizer is an extension tool for Visual Studio. It is a great tool for performance profiling and multithreading execution visualization. It also has a SDK library to be invoked by code and draw markers and spans in the timeline. I used it to visualize sequential LINQ and Parallel LINQ (PLINQ) execution in my Functional Programming and LINQ tutorials. For example, array.Where(…).Select(…) sequential LINQ query and array.AsParallel().Where(…).Select(…) Parallel LINQ query can be visualized as following spans:

The Concurrency Visualizer is available in Visual Studio Marketplace:

And the SDK library is available for .NET Framework, and can be installed from menu item:

This adds a reference to local assembly C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\4hhyuhoo.ghy\SDK\Managed\4.0\Microsoft.ConcurrencyVisualizer.Markers.dll. For your convenience, I have made a NuGet package:

Install-Package ConcurrencyVisualizer

It makes your code more potable. I have also ported the code to .NET Standard, so now the SDK can work with .NET Framework, .NET Core, etc.:

Now the Concurrency Visualizer tool can visualize markers and spans of .NET Core correctly.

TransientFaultHandling.Core: Retry library for .NET Core/.NET Standard

Wed, 22 Jan 2025 00:00:00 GMT

TransientFaultHandling.Core is retry library for transient error handling. It is ported from Microsoft Enterprise Library’s TransientFaultHandling library, a library widely used with .NET Framework. The retry pattern APIs are ported to .NET Core/.NET Standard, with outdated configuration API updated, and new retry APIs added for convenience.

Introduction

With this library, the old code of retry logic based on Microsoft Enterprise Library can be ported to .NET Core/.NET Standard without modification:

ITransientErrorDetectionStrategy transientExceptionDetection = new MyDetection();
RetryStrategy retryStrategy = new FixedInterval(retryCount: 5, retryInterval: TimeSpan.FromSeconds(1));
RetryPolicy retryPolicy = new RetryPolicy(transientExceptionDetection, retryStrategy);
string result = retryPolicy.ExecuteAction(() => webClient.DownloadString("https://DixinYan.com"));

With this library, it is extremely easy to detect transient exception and implement retry logic. For example, the following code downloads a string, if the exception thrown is transient (a WebException), it retries up to 5 times, and it waits for 1 second between retries:

Retry.FixedInterval(
    () => webClient.DownloadString("https://DixinYan.com"),
    isTransient: exception => exception is WebException,
    retryCount: 5, retryInterval: TimeSpan.FromSeconds(1));

Fluent APIs are also provided for even better readability:

Retry
    .WithIncremental(retryCount: 5, initialInterval: TimeSpan.FromSeconds(1),
        increment: TimeSpan.FromSeconds(1))
    .Catch<OperationCanceledException>()
    .Catch<WebException>(exception =>
        exception.Response is HttpWebResponse response && response.StatusCode == HttpStatusCode.RequestTimeout)
    .ExecuteAction(() => webClient.DownloadString("https://DixinYan.com"));

It also supports JSON/XML/INI configuration:

{
  "retryStrategy": {
    "name1": {
      "fastFirstRetry": "true",
      "retryCount": 5,
      "retryInterval": "00:00:00.1"
    },
    "name2": {
      "fastFirstRetry": "true",
      "retryCount": 55,
      "initialInterval": "00:00:00.2",
      "increment": "00:00:00.3"
    }
  }
}

Document

https://weblogs.asp.net/dixin/transientfaulthandling-core-retry-library-for-net-core-net-standard

Source

https://github.com/Dixin/EnterpriseLibrary.TransientFaultHandling.Core (Partially ported from Topaz, with additional new APIs and updated configuration APIs).

NuGet installation

It can be installed through NuGet using .NET CLI:

dotnet add package EnterpriseLibrary.TransientFaultHandling.Core

dotnet add package TransientFaultHandling.Caching

dotnet add package TransientFaultHandling.Configuration

dotnet add package TransientFaultHandling.Data

Or in Visual Studio NuGet Package Manager Console:

Install-Package EnterpriseLibrary.TransientFaultHandling.Core

Install-Package TransientFaultHandling.Caching

Install-Package TransientFaultHandling.Configuration

Install-Package TransientFaultHandling.Data

Backward compatibility with Enterprise Library

This library provides maximum backward compatibility with Microsoft Enterprise Library’s TransientFaultHandling (aka Topaz) for .NET Framework:

If you have code using EnterpriseLibrary.TransientFaultHandling, you can port your code to use EnterpriseLibrary.TransientFaultHandling.Core, without any modification.
If you have code using EnterpriseLibrary.TransientFaultHandling.Caching, you can port your code to use TransientFaultHandling.Caching, without any modification.
If you have code using EnterpriseLibrary.TransientFaultHandling.Data, you can port your code to use TransientFaultHandling.Data, without any modification.
If you have code and configuration based on EnterpriseLibrary.TransientFaultHandling.Configuration, you have to change your code and configuration to use TransientFaultHandling.Configuration. The old XML configuration infrastructure based on .NET Framework is outdated. You need to replace the old XML format with new XML/JSON/INI format configuration supported by .NET Core/.NET Standard.

How to use the APIs

For retry pattern, please read Microsoft’s introduction in Cloud Design Patterns. For the introduction of transient fault handling, read Microsoft’s Perseverance, Secret of All Triumphs: Using the Transient Fault Handling Application Block and Microsoft Azure Architecture Center’s Best practice - Transient fault handling.

Object-oriented APIs from Enterprise Library

Enterprise Library existing APIs follows an object-oriented design. For the details, please see Microsoft’s API reference and ebook Developer's Guide to Microsoft Enterprise Library‘s Chapter 4, Using the Transient Fault Handling Application Block. Here is a brief introduction.

First, ITransientErrorDetectionStrategy interface must be implemented. It has a single method IsTransient to detected if the thrown exception is transient and retry should be executed.

internal class MyDetection : ITransientErrorDetectionStrategy
{
    bool IsTransient(Exception exception) => 
        exception is OperationCanceledException;
}

Second, a retry strategy must be defined to specify how the retry is executed, like retry count, retry interval, etc.. a retry strategy must inherit RetryStrategy abstract class. There are 3 built-in retry strategies: FixedInterval, Incremental, ExponentialBackoff.

Then a retry policy (RetryPolicy class) must be instantiated with a retry strategy and an ITransientErrorDetectionStrategy interface. a retry policy has an ExecuteAction method to execute the specified synchronous function, and an ExecuteAsync method to execute a\the specified async function. It also has a Retrying event. When the executed sync/async function throws an exception, if the exception is detected to be transient and max retry count is not reached, then it waits for the specified retry interval, and then it fires the Retrying event, and execute the specified sync/async function again.

RetryStrategy retryStrategy = new FixedInterval(retryCount: 5, retryInterval: TimeSpan.FromSeconds(1));

RetryPolicy retryPolicy = new RetryPolicy(new MyDetection(), retryStrategy);
retryPolicy.Retrying += (sender, args) =>
    Console.WriteLine($@"{args.CurrentRetryCount}: {args.LastException}");

using (WebClient webClient = new WebClient())
{
    string result1 = retryPolicy.ExecuteAction(() => webClient.DownloadString("https://DixinYan.com"));
    string result2 = await retryPolicy.ExecuteAsync(() => webClient.DownloadStringTaskAsync("https://DixinYan.com"));
}

New functional APIs: single function call for retry

The above object-oriented API design is very inconvenient. New static functions Retry.FixedInterval, Retry.Incremental, Retry.ExponentialBackoff are added to implement retry with a single function call. For example:

Retry.FixedInterval(
    () => webClient.DownloadString("https://DixinYan.com"),
    isTransient: exception => exception is OperationCanceledException,
    retryCount: 5, retryInterval: TimeSpan.FromSeconds(1),
    retryingHandler: (sender, args) =>
        Console.WriteLine($@"{args.CurrentRetryCount}: {args.LastException}"));

await Retry.IncrementalAsync(
    () => webClient.DownloadStringTaskAsync("https://DixinYan.com"),
    isTransient: exception => exception is OperationCanceledException,
    retryCount: 5, initialInterval: TimeSpan.FromSeconds(1), increment: TimeSpan.FromSeconds(2));

These sync and async functions are very convenient because only the first argument (action to execute) is required. All the other arguments are optional. And a function can be defined inline to detect transient exception, instead of defining a type to implement an interface:

// Treat any exception as transient. Use default retry count, default interval. No event handler.
Retry.FixedInterval(() => webClient.DownloadString("https://DixinYan.com"));

// Treat any exception as transient. Specify retry count. Use default initial interval, default increment. No event handler.
await Retry.IncrementalAsync(
    () => webClient.DownloadStringTaskAsync("https://DixinYan.com"),
    retryCount: 10);

New fluent APIs for retry

For better readability, new fluent APIs are provided:

Retry
    .WithFixedInterval(retryCount: 5, retryInterval: TimeSpan.FromSeconds(1))
    .Catch(exception =>
        exception is OperationCanceledException ||
        exception is HttpListenerException httpListenerException && httpListenerException.ErrorCode == 404)
    .HandleWith((sender, args) =>
        Console.WriteLine($@"{args.CurrentRetryCount}: {args.LastException}"))
    .ExecuteAction(() => MyTask());

The HandleWith call adds an event handler to the Retying event. It is optional:

Retry
    .WithFixedInterval(retryCount: 5, retryInterval: TimeSpan.FromSeconds(1))
    .Catch(exception =>
        exception is OperationCanceledException ||
        exception is HttpListenerException httpListenerException && httpListenerException.ErrorCode == 404)
    .ExecuteAction(() => MyTask());

Catch method has a generic overload. The above code is equivalent to:

Retry
    .WithFixedInterval(retryCount: 5, retryInterval: TimeSpan.FromSeconds(1))
    .Catch<OperationCanceledException>()
    .Catch<HttpListenerException>(exception => exception.ErrorCode == 404)
    .ExecuteAction(() => MyTask());

The following code “catches” any exception as transient:

Retry
    .WithIncremental(retryCount: 5, increment: TimeSpan.FromSeconds(1)) // Use default initial interval.
    .Catch() // Equivalent to: .Catch<Exception>()
    .ExecuteAction(() => MyTask());

Old XML configuration for retry

Ditched the following old XML format from .NET Framework:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="RetryPolicyConfiguration" type="Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.Configuration.RetryPolicyConfigurationSettings, Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.Configuration" />
  </configSections>
  <RetryPolicyConfiguration>
    <fixedInterval name="FixedIntervalDefault" maxRetryCount="10" retryInterval="00:00:00.1" />
    <incremental name="IncrementalIntervalDefault" maxRetryCount="10" initialInterval="00:00:00.01" retryIncrement="00:00:00.05" />
    <exponentialBackoff name="ExponentialIntervalDefault" maxRetryCount="10" minBackoff="100" maxBackoff="1000" deltaBackoff="100" />
  </RetryPolicyConfiguration>
</configuration>

These old XML infrastructures are outdated. Use new XML/JSON/INI format configuration supported by .NET Standard/.NET Core.

New XML/JSON/INI configuration for retry

Please install TransientFaultHandling.Configuration package. The following is an example JSON configuration file app.json. It has 3 retry strategies, a FixedInterval retry strategy, a Incremental retry strategy, and an ExponentialBackoff retry strategy:

{
  "retryStrategy": {
    "name1": {
      "fastFirstRetry": "true",
      "retryCount": 5,
      "retryInterval": "00:00:00.1"
    },
    "name2": {
      "fastFirstRetry": "true",
      "retryCount": 55,
      "initialInterval": "00:00:00.2",
      "increment": "00:00:00.3"
    },
    "name3": {
      "fastFirstRetry": "true",
      "retryCount": 555,
      "minBackoff": "00:00:00.4",
      "maxBackoff": "00:00:00.5",
      "deltaBackoff": "00:00:00.6"
    }
  }
}

The same configuration file app.xml in XML format:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <retryStrategy name="name1">
    <fastFirstRetry>true</fastFirstRetry>
    <retryCount>5</retryCount>
    <retryInterval>00:00:00.1</retryInterval>
  </retryStrategy>
  <retryStrategy name="name2">
    <fastFirstRetry>true</fastFirstRetry>
    <retryCount>55</retryCount>
    <initialInterval>00:00:00.2</initialInterval>
    <increment>00:00:00.3</increment>
  </retryStrategy>
  <retryStrategy name="name3">
    <fastFirstRetry>true</fastFirstRetry>
    <retryCount>555</retryCount>
    <minBackoff>00:00:00.4</minBackoff>
    <maxBackoff>00:00:00.5</maxBackoff>
    <deltaBackoff>00:00:00.6</deltaBackoff>
  </retryStrategy>
</configuration>

And app.ini file in INI format:

[retryStrategy:name1]
fastFirstRetry=true
retryCount=5
retryInterval=00:00:00.1

[retryStrategy:name2]
fastFirstRetry=true
retryCount=55
initialInterval=00:00:00.2
increment=00:00:00.3

[retryStrategy:name3]
fastFirstRetry=true
retryCount=5555
minBackoff=00:00:00.4
maxBackoff=00:00:00.5
deltaBackoff=00:00:00.6

These configurations can be easily loaded and deserialized into retry strategy instances:

IConfiguration configuration = new ConfigurationBuilder()
    .AddJsonFile("app.json") // or AddXml("app.xml") or AddIni("app.ini")
    .Build();

IDictionary<string, RetryStrategy> retryStrategies = configuration.GetRetryStrategies();
// or retryStrategies = configuration.GetRetryStrategies("yourConfigurationSectionKey");
// The default configuration section key is "retryStrategy".

The GetRetryStrategies extension method returns a dictionary of key value pairs, where each key is the specified name of retry strategy, and each value is the retry strategy instance. Here the first key is “name1”, the first value is a FixedInterval retry strategy instance. The second key is “anme2”, the second value is Incremental retry strategy instance. The third key is “name3”, the third value is ExponentialBackoff retry strategy instance. This extension method can also accept custom configuration section key, and a function to create instance of custom retry strategy type.

retryStrategies = configuration.GetRetryStrategies(
    key: "yourConfigurationSectionKey",
    getCustomRetryStrategy: configurationSection => new MyRetryStrategyType(...));

The other generic overload can filter the specified retry strategy type:

FixedInterval retryStrategy = configuration.GetRetryStrategies<FixedInterval>().Single().Value;

It still returns a dictionary, which only has the specified type of retry strategies.

TransientFaultHandling.Data.Core: SQL Server support

Since 2.1.0, both Microsoft.Data.SqlClient and System.Data.SqlClient are supported. A API breaking change is introduced for this. If you are using the latest Microsoft.Data.SqlClient, no code change is needed. If you are using the legacy System.Data.SqlClient, the following types are renamed with a Legacy suffix:

ReliableSqlConnection –> ReliableSqlConnectionLegacy
SqlDatabaseTransientErrorDetectionStrategy –> SqlDatabaseTransientErrorDetectionStrategyLegacy
SqlAzureTransientErrorDetectionStrategy –> SqlAzureTransientErrorDetectionStrategyLegacy

You can either rename these types or add the using directives:

using ReliableSqlConnection = Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.ReliableSqlConnectionLegacy;
using SqlDatabaseTransientErrorDetectionStrategy = Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.SqlDatabaseTransientErrorDetectionStrategyLegacy;
using SqlAzureTransientErrorDetectionStrategy = Microsoft.Practices.EnterpriseLibrary.WindowsAzure.TransientFaultHandling.SqlAzure.SqlAzureTransientErrorDetectionStrategyLegacy;

History

This library follows the http://semver.org standard for semantic versioning.

1.0.0: Initial release. Ported EnterpriseLibrary.TransientFaultHandling from .NET Framework to .NET Core/.NET Standard.
1.1.0: Add functional APIs for retry.
1.2.0: Add functional APIs for retry.
2.0.0: Add fluent APIs for retry. Ported EnterpriseLibrary.TransientFaultHandling.Caching from .NET Framework to .NET Core/.NET Standard. Ported EnterpriseLibrary.TransientFaultHandling.Data from .NET Framework to .NET Core/.NET Standard. Redesigned/reimplemented EnterpriseLibrary.TransientFaultHandling.Configuration with JSON in .NET Core/.NET Standard.
2.1.0: Add support for Microsoft.Data.SqlClient. Now both Microsoft.Data.SqlClient and System.Data.SqlClient are supported.

Category Theory via C# (8) Advanced LINQ to Monads

Tue, 31 Dec 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Category Theory via C# series]

Monad is a powerful structure, with the LINQ support in C# language, monad enables chaining operations to build fluent workflow, which can be pure. With these features, monad can be used to manage I/O, state changes, exception handling, shared environment, logging/tracing, and continuation, etc., in the functional paradigm.

IO monad

IO is impure. As already demonstrated, the Lazy<> and Func<> monads can build purely function workflows consists of I/O operations. The I/O is produced only when the workflows is started. So the Func<> monad is also called IO monad (Again, Lazy<T> is just a wrapper of Func<T> factory function, so Lazy<> and Func<> can be viewed as equivalent.). Here, to be more intuitive, rename Func<> to IO<>:

// IO: () -> T
public delegate T IO<out T>();

Func<T> or IO<T> is just a wrapper of T. Generally, the difference is, if a value T is obtained, effect is already produced; and if a Func<T> or IO<T> function wrapper is obtained, the effect can be delayed to produce, until explicitly calling this function to pull the wrapped T value. The following example is a simple comparison:

public static partial class IOExtensions
{
    internal static string Impure()
    {
        string filePath = Console.ReadLine();
        string fileContent = File.ReadAllText(filePath);
        return fileContent;
    }

    internal static IO<string> Pure()
    {
        IO<string> filePath = () => Console.ReadLine();
        IO<string> fileContent = () => File.ReadAllText(filePath());
        return fileContent;
    }

    internal static void IO()
    {
        string ioResult1 = Impure(); // IO is produced.
        IO<string> ioResultWrapper = Pure(); // IO is not produced.

        string ioResult2 = ioResultWrapper(); // IO is produced.
    }
}

IO<> monad is just Func<> monad:

public static partial class IOExtensions
{
    // SelectMany: (IO<TSource>, TSource -> IO<TSelector>, (TSource, TSelector) -> TResult) -> IO<TResult>
    public static IO<TResult> SelectMany<TSource, TSelector, TResult>(
        this IO<TSource> source,
        Func<TSource, IO<TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            () =>
            {
                TSource value = source();
                return resultSelector(value, selector(value)());
            };

    // Wrap: TSource -> IO<TSource>
    public static IO<TSource> IO<TSource>(this TSource value) => () => value;

    // Select: (IO<TSource>, TSource -> TResult) -> IO<TResult>
    public static IO<TResult> Select<TSource, TResult>(
        this IO<TSource> source, Func<TSource, TResult> selector) =>
            source.SelectMany(value => selector(value).IO(), (value, result) => result);
}

The (SelectMany, Wrap, Select) operations are defined so that the LINQ functor syntax (single from clause) and monad syntax (multiple from clauses) are enabled. The let clause is also enabled by Select, which provides great convenience.

Some I/O operations, like above Console.ReadLine: () –> string, and File.ReadAllText: string –> string, returns a value T that can be wrapped IO<T>. There are other I/O operations that return void, like Console.WriteLine: string –> void, etc. Since C# compiler does not allow void to be used as type argument of IO<void>, these operations can be viewed as returning a Unit value, which can be wrapped as IO<Uint>. The following methods help wrap IO<T> functions from I/O operations with or without return value:

public static IO<TResult> IO<TResult>(Func<TResult> function) =>
    () => function();

public static IO<Unit> IO(Action action) =>
    () =>
    {
        action();
        return default;
    };

Now the I/O workflow can be build as purely function LINQ query:

internal static void Workflow()
{
    IO<int> query = from unit1 in IO(() => Console.WriteLine("File path:")) // IO<Unit>.
                    from filePath in IO(Console.ReadLine) // IO<string>.
                    from unit2 in IO(() => Console.WriteLine("File encoding:")) // IO<Unit>.
                    from encodingName in IO(Console.ReadLine) // IO<string>.
                    let encoding = Encoding.GetEncoding(encodingName)
                    from fileContent in IO(() => File.ReadAllText(filePath, encoding)) // IO<string>.
                    from unit3 in IO(() => Console.WriteLine("File content:")) // IO<Unit>.
                    from unit4 in IO(() => Console.WriteLine(fileContent)) // IO<Unit>.
                    select fileContent.Length; // Define query.
    int result = query(); // Execute query.
}

IO<> monad works with both synchronous and asynchronous I/O operations. The async version of IO<T> is just IO<Task<T>>, and the async version of IO<Unit> is just IO<Task>:

internal static async Task WorkflowAsync()
{
    using (HttpClient httpClient = new HttpClient())
    {
        IO<Task> query = from unit1 in IO(() => Console.WriteLine("URI:")) // IO<Unit>. 
                            from uri in IO(Console.ReadLine) // IO<string>.
                            from unit2 in IO(() => Console.WriteLine("File path:")) // IO<Unit>.
                            from filePath in IO(Console.ReadLine) // IO<string>.
                            from downloadStreamTask in IO(async () =>
                                await httpClient.GetStreamAsync(uri)) // IO<Task<Stream>>.
                            from writeFileTask in IO(async () => 
                                await (await downloadStreamTask).CopyToAsync(File.Create(filePath))) // IO<Task>.
                            from messageTask in IO(async () =>
                                {
                                    await writeFileTask;
                                    Console.WriteLine($"Downloaded {uri} to {filePath}");
                                }) // IO<Task>.
                            select messageTask; // Define query.
        await query(); // Execute query.
    }
}

State monad

In object-oriented programming, there is the state pattern to handle state changes. In functional programming, state change can be modeled with pure function. For pure function TSource –> TResult, its state-involved version can be represented as a Tuple<TSource, TState> –> Tuple<TResult, TState> function, which accepts some input value along with some input state, and returns some output value and some output state. This function can remains pure, because it can leave the input state unchanged, then either return the same old state, or create a new state and return it. To make this function monadic, break up the input tuple and curry the function to TSource –> (TState –> Tuple<TResult, TState>). Now the returned TState –> Tuple<TResult, TState> function type can be given an alias called State:

// State: TState -> ValueTuple<T, TState>
public delegate (T Value, TState State) State<TState, T>(TState state);

Similar to fore mentioned Tuple<,> and Func<,> types, the above open generic type State<,> can be viewed as a type constructor of kind * –> * –> *. After partially applied with a first type argument TState, State<TState,> becomes a * –> * type constructor. If it can be a functor and monad, then above stateful function becomes a monadic selector TSource –> State<TState, TResult>. So the following (SelectMany, Wrap, Select) methods can be defined for State<TState,>:

public static partial class StateExtensions
{
    // SelectMany: (State<TState, TSource>, TSource -> State<TState, TSelector>, (TSource, TSelector) -> TResult) -> State<TState, TResult>
    public static State<TState, TResult> SelectMany<TState, TSource, TSelector, TResult>(
        this State<TState, TSource> source,
        Func<TSource, State<TState, TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            oldState =>
            {
                (TSource Value, TState State) value = source(oldState);
                (TSelector Value, TState State) result = selector(value.Value)(value.State);
                TState newState = result.State;
                return (resultSelector(value.Value, result.Value), newState); // Output new state.
            };

    // Wrap: TSource -> State<TState, TSource>
    public static State<TState, TSource> State<TState, TSource>(this TSource value) =>
        oldState => (value, oldState); // Output old state.

    // Select: (State<TState, TSource>, TSource -> TResult) -> State<TState, TResult>
    public static State<TState, TResult> Select<TState, TSource, TResult>(
        this State<TState, TSource> source,
        Func<TSource, TResult> selector) =>
            oldState =>
            {
                (TSource Value, TState State) value = source(oldState);
                TState newState = value.State;
                return (selector(value.Value), newState); // Output new state.
            };
            // Equivalent to:            
            // source.SelectMany(value => selector(value).State<TState, TResult>(), (value, result) => result);
}

SelectMany and Select return a function that accepts an old state and outputs new state, State method returns a function that outputs the old state. Now this State<TState,> delegate type is the state monad, so a State<TState, T> function can be viewed as a wrapper of a T value, and this T value can be unwrapped in the monad workflow, with the from value in source syntax. State<TState, T> function also wraps the state information. To get/set the TState state in the monad workflow, the following GetState/SetState functions can be defined:

// GetState: () -> State<TState, TState>
public static State<TState, TState> GetState<TState>() =>
    oldState => (oldState, oldState); // Output old state.

// SetState: TState -> State<TState, Unit>
public static State<TState, Unit> SetState<TState>(TState newState) =>
    oldState => (default, newState); // Output new state.

Here GetState returns a State<TState, TState> function wrapping the state as value, so that the state can be extracted in the monad workflow with the same syntax that unwraps the value. SetState returns a State<TState, Unit> function, which ignores the old state, and wrap no value (represented by Unit) and outputs the the specified new value to the monad workflow. Generally, the state monad workflow can be demonstrated as:

internal static void Workflow()
{
    string initialState = nameof(initialState);
    string newState = nameof(newState);
    string resetState = nameof(resetState);
    State<string, int> source1 = oldState => (1, oldState);
    State<string, bool> source2 = oldState => (true, newState);
    State<string, char> source3 = '@'.State<string, char>(); // oldState => 2, oldState).

    State<string, string[]> query =
        from value1 in source1 // source1: State<string, int> = initialState => (1, initialState).
        from state1 in GetState<string>() // GetState<int>(): State<string, string> = initialState => (initialState, initialState).
        from value2 in source2 // source2: State<string, bool>3 = initialState => (true, newState).
        from state2 in GetState<string>() // GetState<int>(): State<string, string> = newState => (newState, newState).
        from unit in SetState(resetState) // SetState(resetState): State<string, Unit> = newState => (default, resetState).
        from state3 in GetState<string>() // GetState(): State<string, string> = resetState => (resetState, resetState).
        from value3 in source3 // source3: State<string, char> = resetState => (@, resetState).
        select new string[] { state1, state2, state3 }; // Define query.
    (string[] Value, string State) result = query(initialState); // Execute query with initial state.
    result.Value.WriteLines(); // initialState newState resetState
    result.State.WriteLine(); // Final state: resetState
}

The state monad workflow is a State<TState, T> function, which is of type TState –> Tuple<T, TState>. To execute the workflow, it must be called with a TState initial state. At runtime, when the workflow executes, the first operation in the workflow, also a TState –> Tuple<T, TState> function, is called with the workflow’s initial state, and returns a output value and a output state; then the second operation, once again another TState –> Tuple<T, TState> function, is called with the first operation’s output state, and outputs another output value and another output state; and so on. In this chaining, each operation function can wither return its original input state, or return a new state. This is how state changes through a workflow of pure functions.

Take the factorial function as example. The factorial function can be viewed as a recursive function with a state – the current product of the current recursion step, and apparently take the initial state (product) is 1. To calculate the factorial of 5, the recursive steps can be modeled as:

(Value: 5, State: 1) => (Value: 4, State: 1 * 5)
(Value: 4, State: 1 * 5) => (Value: 3, State: 1 * 5 * 4)
(Value: 3, State: 1 * 5 * 4) => (Value: 3, State: 1 * 5 * 4)
(Value: 2, State: 1 * 5 * 4 * 3) => (Value: 2, State: 1 * 5 * 4 * 3)
(Value: 1, State: 1 * 5 * 4 * 3 * 2) => (Value: 1, State: 1 * 5 * 4 * 3 * 2)
(Value: 0, State: 1 * 5 * 4 * 3 * 2 * 1) => (Value: 0, State: 1 * 5 * 4 * 3 * 2 * 1)

When the current integer becomes 0, the recursion terminates, and the final state (product) is the factorial result. So this recursive function is of type Tuple<int, int> –> Tuple<int, int>. As fore mentioned, it can be curried to int –> (int –> Tuple<int, int>), which is equivalent to int –> State<int, int>:

// FactorialState: uint -> (uint -> (uint, uint))
// FactorialState: uint -> State<unit, uint>
private static State<uint, uint> FactorialState(uint current) =>
    from state in GetState<uint>() // State<uint, uint>.
    let product = state
    let next = current - 1U
    from result in current > 0U
        ? (from unit in SetState(product * current) // State<unit, Unit>.
            from value in FactorialState(next) // State<uint, uint>.
            select next)
        : next.State<uint, uint>() // State<uint, uint>.
    select result;

public static uint Factorial(uint uInt32)
{
    State<uint, uint> query = FactorialState(uInt32); // Define query.
    return query(1).State; // Execute query, with initial state: 1.
}

Another example is Enumerable.Aggregate query method, which accepts an IEnumerable<TSource> sequence, a TAccumulate seed, and a TAccumulate –> TSource –> TAccumulate function. Aggregate calls the accumulation function over the seed and all the values in the sequence. The aggregation steps can also be modeled as recursive steps, where each step’s state is the current accumulate result and the unused source values. Take source sequence { 1, 2, 3, 4, 5 }, seed 0, and function + as example:

(Value: +, State: (0, { 1, 2, 3, 4 })) => (Value: +, State: (0 + 1, { 2, 3, 4 }))
(Value: +, State: (0 + 1, { 2, 3, 4 })) => (Value: +, State: (0 + 1 + 2, { 3, 4 }))
(Value: +, State: (0 + 1 + 2, { 3, 4 })) => (Value: +, State: (0 + 1 + 2 + 3, { 4 }))
(Value: +, State: (0 + 1 + 2 + 3, { 4 })) => (Value: +, State: (0 + 1 + 2 + 3 + 4, { }))
(Value: +, State: (0 + 1 + 2 + 3 + 4, { })) => (Value: +, State: (0 + 1 + 2 + 3 + 4, { }))

When the current source sequence in the state is empty, all source values are applied to the accumulate function, the recursion terminates, and the aggregation result in in the final state. So the recursive function is of type Tuple<TAccumulate –> TSource –> TAccumulate, Tuple<TAccumulate, IEnumerable<TSource>>> –> Tuple<TAccumulate –> TSource –> TAccumulate, Tuple<TAccumulate, IEnumerable<TSource>>>. Again, it can be curried to (TAccumulate –> TSource –> TAccumulate) –> (Tuple<TAccumulate, IEnumerable<TSource>> –> Tuple<TAccumulate –> TSource –> TAccumulate, Tuple<TAccumulate, IEnumerable<TSource>>>), which is equivalent to (TAccumulate –> TSource –> TAccumulate) –> State<Tuple<TAccumulate, IEnumerable<TSource>>, TAccumulate –> TSource –> TAccumulate>:

// AggregateState: (TAccumulate -> TSource -> TAccumulate) -> ((TAccumulate, IEnumerable<TSource>) -> (TAccumulate -> TSource -> TAccumulate, (TAccumulate, IEnumerable<TSource>)))
// AggregateState: TAccumulate -> TSource -> TAccumulate -> State<(TAccumulate, IEnumerable<TSource>), TAccumulate -> TSource -> TAccumulate>
private static State<(TAccumulate, IEnumerable<TSource>), Func<TAccumulate, TSource, TAccumulate>> AggregateState<TSource, TAccumulate>(
    Func<TAccumulate, TSource, TAccumulate> func) =>
        from state in GetState<(TAccumulate, IEnumerable<TSource>)>() // State<(TAccumulate, IEnumerable<TSource>), (TAccumulate, IEnumerable<TSource>)>.
        let accumulate = state.Item1 // TAccumulate.
        let source = state.Item2.Share() // IBuffer<TSource>.
        let sourceIterator = source.GetEnumerator() // IEnumerator<TSource>.
        from result in sourceIterator.MoveNext()
            ? (from unit in SetState((func(accumulate, sourceIterator.Current), source.AsEnumerable())) // State<(TAccumulate, IEnumerable<TSource>), Unit>.
                from value in AggregateState(func) // State<(TAccumulate, IEnumerable<TSource>), Func<TAccumulate, TSource, TAccumulate>>.
                select func)
            : func.State<(TAccumulate, IEnumerable<TSource>), Func<TAccumulate, TSource, TAccumulate>>() // State<(TAccumulate, IEnumerable<TSource>), Func<TAccumulate, TSource, TAccumulate>>.
        select result;

public static TAccumulate Aggregate<TSource, TAccumulate>(
    IEnumerable<TSource> source, TAccumulate seed, Func<TAccumulate, TSource, TAccumulate> func)
{
    State<(TAccumulate, IEnumerable<TSource>), Func<TAccumulate, TSource, TAccumulate>> query =
        AggregateState(func); // Define query.
    return query((seed, source)).State.Item1; // Execute query, with initial state (seed, source).
}

In each recursion step, if the source sequence in the current state in not empty, the source sequence needs to be split. The first value is used to call the accumulation function, and the other values are put into output state, which is passed to the next recursion step. So there are multiple pulling operations for the source sequence: detecting if it is empty detection, pulling first value, and pulling the rest values. To avoid multiple iterations for the same source sequence, here the Share query method from Microsoft Ix (Interactive Extensions) library is called, so that all the pulling operations share the same iterator.

The stack’s Pop and Push operation can be also viewed as state processing. The Pop method of stack requires no input, and out put the stack’s top value T, So Pop can be viewed of type Unit –> T. In contrast, stack’s Push method accepts a value, set the value to the top of the stack, and returns no output, so Push can be viewed of type T –> Unit. The stack’s values are different before and after the Pop and Push operations, so the stack itself can be viewed as the state of the Pop and Push operation. If the values in a stack is represented as a IEnumerable<T> sequence, then Pop can be remodeled as Tuple<Unit, IEnumerable<T>> –> Tuple<Unit, IEnumerable<T>>, which can be curried to Unit –> State<IEnumerable<T>, T>; and Push can be remodeled as Tuple<T, IEnumerable<T>> –> Tuple<Unit, IEnumerable<T>>:

// PopState: Unit -> (IEnumerable<T> -> (T, IEnumerable<T>))
// PopState: Unit -> State<IEnumerable<T>, T>
internal static State<IEnumerable<T>, T> PopState<T>(Unit unit = null) =>
    oldStack =>
    {
        IEnumerable<T> newStack = oldStack.Share();
        return (newStack.First(), newStack); // Output new state.
    };

// PushState: T -> (IEnumerable<T> -> (Unit, IEnumerable<T>))
// PushState: T -> State<IEnumerable<T>, Unit>
internal static State<IEnumerable<T>, Unit> PushState<T>(T value) =>
    oldStack =>
    {
        IEnumerable<T> newStack = oldStack.Concat(value.Enumerable());
        return (default, newStack); // Output new state.
    };

Now the stack operations can be a state monad workflow. Also, GetState can get the current values of the stack, and SetState can reset the values of stack:

internal static void Stack()
{
    IEnumerable<int> initialStack = Enumerable.Repeat(0, 5);
    State<IEnumerable<int>, IEnumerable<int>> query =
        from value1 in PopState<int>() // State<IEnumerable<int>, int>.
        from unit1 in PushState(1) // State<IEnumerable<int>, Unit>.
        from unit2 in PushState(2) // State<IEnumerable<int>, Unit>.
        from stack in GetState<IEnumerable<int>>() // State<IEnumerable<int>, IEnumerable<int>>.
        from unit3 in SetState(Enumerable.Range(0, 5)) // State<IEnumerable<int>, Unit>.
        from value2 in PopState<int>() // State<IEnumerable<int>, int>.
        from value3 in PopState<int>() // State<IEnumerable<int>, int>.
        from unit4 in PushState(5) // State<IEnumerable<int>, Unit>.
        select stack; // Define query.
    (IEnumerable<int> Value, IEnumerable<int> State) result = query(initialStack); // Execute query with initial state.
    result.Value.WriteLines(); // 0 0 0 0 1 2
    result.State.WriteLines(); // 0 1 2 5
}

Exception monad

As previously demonstrated, the Optional<> monad can handle the case that any operation of the workflow may not produce a valid result, in a . When an operation succeeds to return a valid result, the next operation executes. If all operations succeed, the entire workflow has a valid result. Option<> monad’s handling is based on operation’s return result. What if the operation fails with exception? To work with operation exceptions in a purely functional paradigm, the following Try<> structure can be defined, which is just Optional<> plus exception handling and store:

public readonly struct Try<T>
{
    private readonly Lazy<(T, Exception)> factory;

    public Try(Func<(T, Exception)> factory) =>
        this.factory = new Lazy<(T, Exception)>(() =>
        {
            try
            {
                return factory();
            }
            catch (Exception exception)
            {
                return (default, exception);
            }
        });

    public T Value
    {
        get
        {
            if (this.HasException)
            {
                throw new InvalidOperationException($"{nameof(Try<T>)} object must have a value.");
            }
            return this.factory.Value.Item1;
        }
    }

    public Exception Exception => this.factory.Value.Item2;

    public bool HasException => this.Exception != null;

    public static implicit operator Try<T>(T value) => new Try<T>(() => (value, (Exception)null));
}

Try<T> represents an operation, which either succeeds with a result, or fail with an exception. Its SelectMany method is also in the same pattern as Optional<>’s SelectMany, so that when an operation (source) succeeds without exception, the next operation (returned by selector) executes:

public static partial class TryExtensions
{
    // SelectMany: (Try<TSource>, TSource -> Try<TSelector>, (TSource, TSelector) -> TResult) -> Try<TResult>
    public static Try<TResult> SelectMany<TSource, TSelector, TResult>(
        this Try<TSource> source,
        Func<TSource, Try<TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            new Try<TResult>(() =>
            {
                if (source.HasException)
                {
                    return (default, source.Exception);
                }
                Try<TSelector> result = selector(source.Value);
                if (result.HasException)
                {
                    return (default, result.Exception);
                }
                return (resultSelector(source.Value, result.Value), (Exception)null);
            });

    // Wrap: TSource -> Try<TSource>
    public static Try<TSource> Try<TSource>(this TSource value) => value;

    // Select: (Try<TSource>, TSource -> TResult) -> Try<TResult>
    public static Try<TResult> Select<TSource, TResult>(
        this Try<TSource> source, Func<TSource, TResult> selector) =>
            source.SelectMany(value => selector(value).Try(), (value, result) => result);
}

The operation of throwing an exception can be represented with a Try<T> with the specified exception:

public static Try<T> Throw<T>(this Exception exception) => new Try<T>(() => (default, exception));

For convenience, Try<T> instance can be implicitly wrapped from a T value. And the following method also helps wrap a Func<T> operation:

public static Try<T> Try<T>(Func<T> function) =>
    new Try<T>(() => (function(), (Exception)null));

Similar to IO<> monad, an function operation (() –> void) without return result can be viewed as a function returning Unit (() –> Unit):

public static Try<Unit> Try(Action action) =>
    new Try<Unit>(() =>
    {
        action();
        return (default, (Exception)null);
    });

To handle the exception from an operation represented by Try<T>, just check the HasException property, filter the exception, and process it. The following Catch method handles the specified exception type:

public static Try<T> Catch<T, TException>(
    this Try<T> source, Func<TException, Try<T>> handler, Func<TException, bool> when = null)
    where TException : Exception => 
        new Try<T>(() =>
        {
            if (source.HasException && source.Exception is TException exception && exception != null
                && (when == null || when(exception)))
            {
                source = handler(exception);
            }
            return source.HasException ? (default, source.Exception) : (source.Value, (Exception)null);
        });

The evaluation of the Try<T> source, and the execution of handler, are both deferred. And the following Catch overload handles all exception types:

public static Try<T> Catch<T>(
    this Try<T> source, Func<Exception, Try<T>> handler, Func<Exception, bool> when = null) =>
        Catch<T, Exception>(source, handler, when);

And the Finally method just call a function to process the Try<T>:

public static TResult Finally<T, TResult>(
    this Try<T> source, Func<Try<T>, TResult> finally) => finally(source);

public static void Finally<T>(
    this Try<T> source, Action<Try<T>> finally) => finally(source);

The operation of throwing an exception can be represented with a Try<T> instance wrapping the specified exception:

public static partial class TryExtensions
{
    public static Try<T> Throw<T>(this Exception exception) => new Try<T>(() => (default, exception));
}

The following is an example of throwing exception:

internal static Try<int> TryStrictFactorial(int? value)
{
    if (value == null)
    {
        return Throw<int>(new ArgumentNullException(nameof(value)));
    }
    if (value <= 0)
    {
        return Throw<int>(new ArgumentOutOfRangeException(nameof(value), value, "Argument should be positive."));
    }

    if (value == 1)
    {
        return 1;
    }
    return value.Value * TryStrictFactorial(value - 1).Value;
}

And the the following is an example of handling exception:

internal static string Factorial(string value)
{
    Func<string, int?> stringToNullableInt32 = @string =>
        string.IsNullOrEmpty(@string) ? default : Convert.ToInt32(@string);
    Try<int> query = from nullableInt32 in Try(() => stringToNullableInt32(value)) // Try<int32?>
                        from result in TryStrictFactorial(nullableInt32) // Try<int>.
                        from unit in Try(() => result.WriteLine()) // Try<Unit>.
                        select result; // Define query.
    return query
        .Catch(exception => // Catch all and rethrow.
        {
            exception.WriteLine();
            return Throw<int>(exception);
        })
        .Catch<int, ArgumentNullException>(exception => 1) // When argument is null, factorial is 1.
        .Catch<int, ArgumentOutOfRangeException>(
            when: exception => object.Equals(exception.ActualValue, 0),
            handler: exception => 1) // When argument is 0, factorial is 1.
        .Finally(result => result.HasException // Execute query.
            ? result.Exception.Message : result.Value.ToString());
}

Reader monad

The Func<T,> functor is also monad. In contrast to Func<> monad, a factory function that only outputs a value, Func<T,> can also read input value from the environment. So Fun<T,> monad is also called reader monad, or environment monad. To be intuitive, rename Func<T,> to Reader<TEnvironment,>:

// Reader: TEnvironment -> T
public delegate T Reader<in TEnvironment, out T>(TEnvironment environment);

And its (SelectMany, Wrap, Select) methods are straightforward:

public static partial class ReaderExtensions
{
    // SelectMany: (Reader<TEnvironment, TSource>, TSource -> Reader<TEnvironment, TSelector>, (TSource, TSelector) -> TResult) -> Reader<TEnvironment, TResult>
    public static Reader<TEnvironment, TResult> SelectMany<TEnvironment, TSource, TSelector, TResult>(
        this Reader<TEnvironment, TSource> source,
        Func<TSource, Reader<TEnvironment, TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            environment =>
            {
                TSource value = source(environment);
                return resultSelector(value, selector(value)(environment));
            };

    // Wrap: TSource -> Reader<TEnvironment, TSource>
    public static Reader<TEnvironment, TSource> Reader<TEnvironment, TSource>(this TSource value) => 
        environment => value;

    // Select: (Reader<TEnvironment, TSource>, TSource -> TResult) -> Reader<TEnvironment, TResult>
    public static Reader<TEnvironment, TResult> Select<TEnvironment, TSource, TResult>(
        this Reader<TEnvironment, TSource> source, Func<TSource, TResult> selector) =>
            source.SelectMany(value => selector(value).Reader<TEnvironment, TResult>(), (value, result) => result);
}

There are scenarios of accessing input value from shared environment, like reading the configurations, dependency injection, etc. In the following example, the operations are dependents of the configurations, so these operations can be modeled using Reader<ICongiguration,> monad:

private static Reader<IConfiguration, FileInfo> DownloadHtml(Uri uri) =>
    configuration => default;

private static Reader<IConfiguration, FileInfo> ConverToWord(FileInfo htmlDocument, FileInfo template) =>
    configuration => default;

private static Reader<IConfiguration, Unit> UploadToOneDrive(FileInfo file) =>
    configuration => default;

internal static void Workflow(IConfiguration configuration, Uri uri, FileInfo template)
{
    Reader<IConfiguration, (FileInfo, FileInfo)> query =
        from htmlDocument in DownloadHtml(uri) // Reader<IConfiguration, FileInfo>.
        from wordDocument in ConverToWord(htmlDocument, template) // Reader<IConfiguration, FileInfo>.
        from unit in UploadToOneDrive(wordDocument) // Reader<IConfiguration, Unit>.
        select (htmlDocument, wordDocument); // Define query.
    (FileInfo, FileInfo) result = query(configuration); // Execute query.
}

The workflow is also a Reader<ICongiguration, T> function. To execute the workflow, it must read the required configuration input. Then all operation in the workflow execute sequentially by reading the same configuration input.

Writer monad

Writer is a function that returns a computed value along with a stream of additional content, so this function is of type () –> Tuple<T, TContent>. In the writer monad workflow, each operation’s additional output content is merged with the next operation’s additional output content, so that when the entire workflow is executed, all operations’ additional output content are merged as the workflow’s final additional output content. Each merge operation accepts 2 TContent instances, and result another TContent instance. It is a binary operation and can be implemented by monoid’s multiplication: TContent ⊙ TContent –> TContent. So writer can be represented by a () –> Tuple<T, TContent> function along with a IMonoid<TContent> monoid:

public abstract class WriterBase<TContent, T, TContentMonoid> : IMonoid<TContent> where TContentMonoid : IMonoid<TContent>
{
    private readonly Lazy<(TContent, T)> lazy;

    protected WriterBase(Func<(TContent, T)> writer)
    {
        this.lazy = new Lazy<(TContent, T)>(writer);
    }

    public TContent Content => this.lazy.Value.Item1;

    public T Value => this.lazy.Value.Item2;

    public static TContent Multiply(TContent value1, TContent value2) => TContentMonoid.Multiply(value1, value2);

    public static TContent Unit => TContentMonoid.Unit;
}

The most common scenario of outputting additional content, is tracing and logging, where the TContent is a sequence of log entries. A sequence of log entries can be represented as IEnumerable<T>, so the fore mentioned (IEnumerable<T>, Enumerable.Concat<T>, Enumerable.Empty<T>()) monoid can be used:

public class Writer<TEntry, T> : WriterBase<IEnumerable<TEntry>, T, EnumerableConcatMonoid<TEntry>>
{
    public Writer(Func<(IEnumerable<TEntry>, T)> writer) : base(writer) { }

    public Writer(T value) : base(() => (Unit, value)) { }
}

Similar to State<TState,> and Reader<TEnvironment,>, here Writer<TEntry,> can be monad with the following (SelectMany, Wrap, Select) methods:

public static partial class WriterExtensions
{
    // SelectMany: (Writer<TEntry, TSource>, TSource -> Writer<TEntry, TSelector>, (TSource, TSelector) -> TResult) -> Writer<TEntry, TResult>
    public static Writer<TEntry, TResult> SelectMany<TEntry, TSource, TSelector, TResult>(
        this Writer<TEntry, TSource> source,
        Func<TSource, Writer<TEntry, TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            new Writer<TEntry, TResult>(() =>
            {
                Writer<TEntry, TSelector> result = selector(source.Value);
                return (Tutorial.CategoryTheory.Writer<TEntry, TSource>.Multiply(source.Content, result.Content),
                    resultSelector(source.Value, result.Value));
            });

    // Wrap: TSource -> Writer<TEntry, TSource>
    public static Writer<TEntry, TSource> Writer<TEntry, TSource>(this TSource value) =>
        new Writer<TEntry, TSource>(value);

    // Select: (Writer<TEnvironment, TSource>, TSource -> TResult) -> Writer<TEnvironment, TResult>
    public static Writer<TEntry, TResult> Select<TEntry, TSource, TResult>(
        this Writer<TEntry, TSource> source, Func<TSource, TResult> selector) =>
            source.SelectMany(value => selector(value).Writer<TEntry, TResult>(), (value, result) => result);
}

Most commonly, each operation in the workflow logs string message. So the following method is defined to construct a writer instance from a value and a string log factory:

public static Writer<string, TSource> LogWriter<TSource>(this TSource value, Func<TSource, string> logFactory) =>
    new Writer<string, TSource>(() => (logFactory(value).Enumerable(), value));

The previous Fun<> monad workflow now can output logs for each operation:

internal static void Workflow()
{
    Writer<string, string> query = from filePath in Console.ReadLine().LogWriter(value =>
                                        $"File path: {value}") // Writer<string, string>.
                                   from encodingName in Console.ReadLine().LogWriter(value =>
                                        $"Encoding name: {value}") // Writer<string, string>.
                                   from encoding in Encoding.GetEncoding(encodingName).LogWriter(value =>
                                        $"Encoding: {value}") // Writer<string, Encoding>.
                                   from fileContent in File.ReadAllText(filePath, encoding).LogWriter(value =>
                                        $"File content length: {value.Length}") // Writer<string, string>.
                                   select fileContent; // Define query.
    string result = query.Value; // Execute query.
    query.Content.WriteLines();
    // File path: D:\File.txt
    // Encoding name: utf-8
    // Encoding: System.Text.UTF8Encoding
    // File content length: 76138
}

Continuation monad

In program, a function can return the result value, so that some other continuation function can use that value; or a function can take a continuation function as parameter, after it computes the result value, it calls back the continuation function with that value:

public static partial class CpsExtensions
{
    // Sqrt: int -> double
    internal static double Sqrt(int int32) => Math.Sqrt(int32);

    // SqrtWithCallback: (int, double -> TContinuation) -> TContinuation
    internal static TContinuation SqrtWithCallback<TContinuation>(
        int int32, Func<double, TContinuation> continuation) =>
            continuation(Math.Sqrt(int32));
}

The former is style is called direct style, and the latter is called continuation-passing style (CPS). Generally, for a TSource –> TResult function, its CPS version can accept a TResult –> TContinuation continuation function, so the CPS function is of type (TSource, TResult –> TContinuation) –> TContinuation. Again, just like the state monad, the CPS function can be curried to TSource –> ((TResult –> TContinuation) –> TContinuation)

// SqrtWithCallback: int -> (double -> TContinuation) -> TContinuation
internal static Func<Func<double, TContinuation>, TContinuation> SqrtWithCallback<TContinuation>(int int32) =>
    continuation => continuation(Math.Sqrt(int32));

Now the returned (TResult –> TContinuation) –> TContinuation function type can be given an alias Cps:

// Cps: (T -> TContinuation>) -> TContinuation
public delegate TContinuation Cps<TContinuation, out T>(Func<T, TContinuation> continuation);

So that the above function can be renamed as:

// SqrtCps: int -> Cps<TContinuation, double>
internal static Cps<TContinuation, double> SqrtCps<TContinuation>(int int32) =>
    continuation => continuation(Math.Sqrt(int32));

The CPS function becomes TSource –> Cps<TContinuation, TResult>, which is a monadic selector function. Just like State<TState,>, here Cps<TContinuation,> is the continuation monad. Its (SelectMany, Wrap, Select) methods can be implemented as:

public static partial class CpsExtensions
{
    // SelectMany: (Cps<TContinuation, TSource>, TSource -> Cps<TContinuation, TSelector>, (TSource, TSelector) -> TResult) -> Cps<TContinuation, TResult>
    public static Cps<TContinuation, TResult> SelectMany<TContinuation, TSource, TSelector, TResult>(
        this Cps<TContinuation, TSource> source,
        Func<TSource, Cps<TContinuation, TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            continuation => source(value =>
                selector(value)(result =>
                    continuation(resultSelector(value, result))));

    // Wrap: TSource -> Cps<TContinuation, TSource>
    public static Cps<TContinuation, TSource> Cps<TContinuation, TSource>(this TSource value) =>
        continuation => continuation(value);

    // Select: (Cps<TContinuation, TSource>, TSource -> TResult) -> Cps<TContinuation, TResult>
    public static Cps<TContinuation, TResult> Select<TContinuation, TSource, TResult>(
        this Cps<TContinuation, TSource> source, Func<TSource, TResult> selector) =>
            source.SelectMany(value => selector(value).Cps<TContinuation, TResult>(), (value, result) => result);
            // Equivalent to:
            // continuation => source(value => continuation(selector(value)));
            // Or:
            // continuation => source(continuation.o(selector));
}

A more complex example is sum of squares. The CPS version of sum and square are straightforward. If direct style of square operation of type int –> int, and the direct style of sum operation is (int, int) –> int, then their CPS versions are just of type int –> Cps<TContinuation, int>, and (int, int) –> Cps<TContinuation, int>:

// SquareCps: int -> Cps<TContinuation, int>
internal static Cps<TContinuation, int> SquareCps<TContinuation>(int x) =>
    continuation => continuation(x * x);

// SumCps: (int, int) -> Cps<TContinuation, int>
internal static Cps<TContinuation, int> SumCps<TContinuation>(int x, int y) =>
    continuation => continuation(x + y);

Then CPS version of sum of square can be implemented with them:

// SumOfSquaresCps: (int, int) -> Cps<TContinuation, int>
internal static Cps<TContinuation, int> SumOfSquaresCps<TContinuation>(int a, int b) =>
    continuation =>
        SquareCps<TContinuation>(a)(squareOfA =>
        SquareCps<TContinuation>(b)(squareOfB =>
        SumCps<TContinuation>(squareOfA, squareOfB)(continuation)));

This is not intuitive. But the continuation monad can help. A Cps<TContinuation, T> function can be viewed as a monad wrapper of T value. So T value can be unwrapped from Cps<TContinuation, T> with the LINQ from clause:

internal static Cps<TContinuation, int> SumOfSquaresCpsLinq<TContinuation>(int a, int b) =>
    from squareOfA in SquareCps<TContinuation>(a) // Cps<TContinuation, int>.
    from squareOfB in SquareCps<TContinuation>(b) // Cps<TContinuation, int>.
    from sum in SumCps<TContinuation>(squareOfA, squareOfB) // Cps<TContinuation, int>.
    select sum;

And the following is a similar example of fibonacci:

internal static Cps<TContinuation, uint> FibonacciCps<TContinuation>(uint uInt32) =>
    uInt32 > 1
        ? (from a in FibonacciCps<TContinuation>(uInt32 - 1U)
            from b in FibonacciCps<TContinuation>(uInt32 - 2U)
            select a + b)
        : uInt32.Cps<TContinuation, uint>();
    // Equivalent to:
    // continuation => uInt32 > 1U
    //    ? continuation(FibonacciCps<int>(uInt32 - 1U)(Id) + FibonacciCps<int>(uInt32 - 2U)(Id))
    //    : continuation(uInt32);

Generally, a direct style function can be easily converted to CPS function – just pass the direct style function’s return value to a continuation function:

public static Cps<TContinuation, T> Cps<TContinuation, T>(Func<T> function) =>
    continuation => continuation(function());

Now the previous workflows can be represented in CPS too:

internal static void Workflow<TContinuation>(Func<string, TContinuation> continuation)
{
    Cps<TContinuation, string> query =
        from filePath in Cps<TContinuation, string>(Console.ReadLine) // Cps<TContinuation, string>.
        from encodingName in Cps<TContinuation, string>(Console.ReadLine) // Cps<TContinuation, string>.
        from encoding in Cps<TContinuation, Encoding>(() => Encoding.GetEncoding(encodingName)) // Cps<TContinuation, Encoding>.
        from fileContent in Cps<TContinuation, string>(() => File.ReadAllText(filePath, encoding)) // Cps<TContinuation, string>.
        select fileContent; // Define query.
    TContinuation result = query(continuation); // Execute query.
}

In the workflow, each operation’s continuation function is its next operation. When the workflow executes, each operation computes its return value, then calls back its next operation with its return value. When the last operation executes, it calls back the workflow’s continuation function.

Category Theory via C# (7) Monad and LINQ to Monads

Thu, 26 Dec 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Category Theory via C# series]

Monad

As fore mentioned endofunctor category can be monoidal (the entire category. Actually, an endofunctor In the endofunctor category can be monoidal too. This kind of endofunctor is called monad. Monad is another important algebraic structure in category theory and LINQ. Formally, monad is an endofunctor equipped with 2 natural transformations:

Monoid multiplication ◎ or μ, which a natural transformation ◎: F(F) ⇒ F, which means, for each object X, ◎ maps F(F(X)) to F(X). For convenience, this mapping operation is also denoted F ◎ F ⇒ F.
Monoid unit η, which is a natural transformation η: I ⇒ F. Here I is the identity functor, which maps each object X to X itself. For each X, there is η maps I(X) to F(X). Since I(X) is just X, η can also be viewed as mapping: X → F(X).

So monad F is a monoid (F, ◎, η) in the category of endofunctors. Apparently it must preserve the monoid laws:

Associativity preservation α: (F ◎ F) ◎ F ≡ F ◎ (F ◎ F)
Left unit preservation λ: η ◎ F ≡ F, and right unit preservation ρ: F ≡ F ◎ η

So that, the following diagram commutes:

In DotNet category, monad can be defined as:

// Cannot be compiled.
public partial interface IMonad<TMonad<>> : IFunctor<TMonad<>> where TMonad<> : IMonad<TMonad<>>
{
    // From IFunctor<TMonad<>>:
    // Select: (TSource -> TResult) -> (TMonad<TSource> -> TMonad<TResult>)
    // Func<TMonad<TSource>, TMonad<TResult>> Select<TSource, TResult>(Func<TSource, TResult> selector);

    // Multiply: TMonad<TMonad<TSource>> -> TMonad<TSource>
    TMonad<TSource> Multiply<TSource>(TMonad<TMonad<TSource>> sourceWrapper);
        
    // Unit: TSource -> TMonad<TSource>
    TMonad<TSource> Unit<TSource>(TSource value);
}

LINQ to Monads and monad laws

Built-in IEnumerable<> monad

The previously discussed IEnumerable<> functor is a built-in monad, it is straightforward to implement its (Multiply, Unit) method pair:

public static partial class EnumerableExtensions // IEnumerable<T> : IMonad<IEnumerable<>>
{
    // Multiply: IEnumerable<IEnumerable<TSource>> -> IEnumerable<TSource>
    public static IEnumerable<TSource> Multiply<TSource>(this IEnumerable<IEnumerable<TSource>> sourceWrapper)
    {
        foreach (IEnumerable<TSource> source in sourceWrapper)
        {
            foreach (TSource value in source)
            {
                yield return value;
            }
        }
    }

    // Unit: TSource -> IEnumerable<TSource>
    public static IEnumerable<TSource> Unit<TSource>(TSource value)
    {
        yield return value;
    }
}

The monoid unit η is exactly the same as the Wrap method for monoidal functor. It is easy to verify the above implementation preserves the monoid laws:

internal static void MonoidLaws()
{
    IEnumerable<int> source = new int[] { 0, 1, 2, 3, 4 };

    // Associativity preservation: source.Wrap().Multiply().Wrap().Multiply() == source.Wrap().Wrap().Multiply().Multiply().
    source.Enumerable().Multiply().Enumerable().Multiply().WriteLines();
    // 0 1 2 3 4
    source.Enumerable().Enumerable().Multiply().Multiply().WriteLines();
    // 0 1 2 3 4
    // Left unit preservation: Unit(source).Multiply() == f.
    Unit(source).Multiply().WriteLines(); // 0 1 2 3 4
    // Right unit preservation: source == source.Select(Unit).Multiply().
    source.Select(Unit).Multiply().WriteLines(); // 0 1 2 3 4
}

As discussed in LINQ to Object chapter, for IEnumerable<>, there is already a query method SelectMany providing the same ability to flatten hierarchy an IEnumerable<IEnumerable<T>> sequence to an IEnumerable<T> sequence. Actually, monad can be alternatively defined with SelectMany and η/Wrap:

public partial interface IMonad<TMonad> where TMonad<> : IMonad<TMonad<>>
{
    // SelectMany: (TMonad<TSource>, TSource -> TMonad<TSelector>, (TSource, TSelector) -> TResult) -> TMonad<TResult>
    TMonad<TResult> SelectMany<TSource, TSelector, TResult>(
        TMonad<TSource> source,
        Func<TSource, TMonad<TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector);

    // Wrap: TSource -> IEnumerable<TSource>
    TMonad<TSource> Wrap<TSource>(TSource value);
}

And the alternative implementation is very similar:

public static partial class EnumerableExtensions // IEnumerable<T> : IMonad<IEnumerable<>>
{
    // SelectMany: (IEnumerable<TSource>, TSource -> IEnumerable<TSelector>, (TSource, TSelector) -> TResult) -> IEnumerable<TResult>
    public static IEnumerable<TResult> SelectMany<TSource, TSelector, TResult>(
        this IEnumerable<TSource> source,
        Func<TSource, IEnumerable<TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector)
    {
        foreach (TSource value in source)
        {
            foreach (TSelector result in selector(value))
            {
                yield return resultSelector(value, result);
            }
        }
    }

    // Wrap: TSource -> IEnumerable<TSource>
    public static IEnumerable<TSource> Enumerable<TSource>(this TSource value)
    {
        yield return value;
    }
}

The above 2 versions of monad definition are equivalent. First, the (SelectMany, Wrap) methods can be implemented with the (Select, Multiply, Unit) methods:

public static partial class EnumerableExtensions // (Select, Multiply, Unit) implements (SelectMany, Wrap).
{
    // SelectMany: (IEnumerable<TSource>, TSource -> IEnumerable<TSelector>, (TSource, TSelector) -> TResult) -> IEnumerable<TResult>
    public static IEnumerable<TResult> SelectMany<TSource, TSelector, TResult>(
        this IEnumerable<TSource> source,
        Func<TSource, IEnumerable<TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            (from value in source
             select (from result in selector(value)
                     select resultSelector(value, result))).Multiply();
            // Compiled to:
            // source.Select(value => selector(value).Select(result => resultSelector(value, result))).Multiply();

    // Wrap: TSource -> IEnumerable<TSource>
    public static IEnumerable<TSource> Enumerable<TSource>(this TSource value) => Unit(value);
}

And the (Select, Multiply, Unit) methods can be implemented with (SelectMany, Wrap) methods too:

public static partial class EnumerableExtensions // (SelectMany, Wrap) implements (Select, Multiply, Unit).
{
    // Select: (TSource -> TResult) -> (IEnumerable<TSource> -> IEnumerable<TResult>).
    public static Func<IEnumerable<TSource>, IEnumerable<TResult>> Select<TSource, TResult>(
        Func<TSource, TResult> selector) => source =>
            from value in source
            from result in value.Enumerable()
            select result;
            // source.SelectMany(Enumerable, (result, value) => value);

    // Multiply: IEnumerable<IEnumerable<TSource>> -> IEnumerable<TSource>
    public static IEnumerable<TSource> Multiply<TSource>(this IEnumerable<IEnumerable<TSource>> sourceWrapper) =>
        from source in sourceWrapper
        from value in source
        select value;
        // sourceWrapper.SelectMany(source => source, (source, value) => value);

    // Unit: TSource -> IEnumerable<TSource>
    public static IEnumerable<TSource> Unit<TSource>(TSource value) => value.Enumerable();
}

So monad support is built-in in the C# language. As discussed in the LINQ query expression pattern part, SelectMany enables multiple from clauses, which can chain operations together to build a workflow, for example:

internal static void Workflow<T1, T2, T3, T4>(
    Func<IEnumerable<T1>> source1,
    Func<IEnumerable<T2>> source2,
    Func<IEnumerable<T3>> source3,
    Func<T1, T2, T3, IEnumerable<T4>> source4)
{
    IEnumerable<T4> query = from value1 in source1()
                            from value2 in source2()
                            from value3 in source3()
                            from value4 in source4(value1, value2, value3)
                            select value4; // Define query.
    query.WriteLines(); // Execute query.
}

Here N + 1 from clauses are compiled to N SelectMany fluent calls:

internal static void CompiledWorkflow<T1, T2, T3, T4>(
    Func<IEnumerable<T1>> source1,
    Func<IEnumerable<T2>> source2,
    Func<IEnumerable<T3>> source3,
    Func<T1, T2, T3, IEnumerable<T4>> source4)
{
    IEnumerable<T4> query = source1()
        .SelectMany(value1 => source2(), (value1, value2) => new { Value1 = value1, Value2 = value2 })
        .SelectMany(result2 => source3(), (result2, value3) => new { Result2 = result2, Value3 = value3 })
        .SelectMany(
            result3 => source4(result3.Result2.Value1, result3.Result2.Value2, result3.Value3),
            (result3, value4) => value4); // Define query.
    query.WriteLines(); // Execute query.
}

In LINQ, if monad’s SelectMany implements deferred execution, then monad enables imperative programming paradigm (a sequence of commands) in a purely functional way. In above LINQ query definition, the calls to the commands are not executed. When trying to pull results from the LINQ query, the workflow stars, and the commands executes sequentially.

Monad law and Kleisli composition

Regarding monad (F, ◎, η) can be redefined as (F, SelectMany, Wrap), the monoid laws now can be expressed by SelectMany and Wrap too, which are called monad laws:

Associativity law: SelectMany is the associative operator, since it is equivalent to Multiply.
Left unit law and right unit law: Wrap is the unit η, since it is identical to Unit.

internal static void MonadLaws()
{
    IEnumerable<int> source = new int[] { 0, 1, 2, 3, 4 };
    Func<int, IEnumerable<char>> selector = int32 => new string('*', int32);
    Func<int, IEnumerable<double>> selector1 = int32 => new double[] { int32 / 2D, Math.Sqrt(int32) };
    Func<double, IEnumerable<string>> selector2 =
        @double => new string[] { @double.ToString("0.0"), @double.ToString("0.00") };
    const int Value = 5;

    // Associativity: source.SelectMany(selector1).SelectMany(selector2) == source.SelectMany(value => selector1(value).SelectMany(selector2)).
    (from value in source
     from result1 in selector1(value)
     from result2 in selector2(result1)
     select result2).WriteLines();
    // 0.0 0.00 0.0 0.00
    // 0.5 0.50 1.0 1.00
    // 1.0 1.00 1.4 1.41
    // 1.5 1.50 1.7 1.73
    // 2.0 2.00 2.0 2.00
    (from value in source
     from result in (from result1 in selector1(value)
                     from result2 in selector2(result1)
                     select result2)
     select result).WriteLines();
    // 0.0 0.00 0.0 0.00
    // 0.5 0.50 1.0 1.00
    // 1.0 1.00 1.4 1.41
    // 1.5 1.50 1.7 1.73
    // 2.0 2.00 2.0 2.00
    // Left unit: value.Wrap().SelectMany(selector) == selector(value).
    (from value in Value.Enumerable()
     from result in selector(value)
     select result).WriteLines(); // * * * * *
    selector(Value).WriteLines(); // * * * * *
    // Right unit: source == source.SelectMany(Wrap).
    (from value in source
     from result in value.Enumerable()
     select result).WriteLines(); // 0 1 2 3 4
}

However, the monad laws are not intuitive. The Kleisli composition ∘ can help. For 2 monadic selector functions that can be passed to SelectMany,are also called Kleisli functions like s1: TSource –> TMonad<TMiddle> and s2: TMiddle –> TMonad<TResult>, their Kleisli composition is still a monadic selector (s2 ∘ s1): TSource –> TMonad<TResult>:

public static Func<TSource, IEnumerable<TResult>> o<TSource, TMiddle, TResult>( // After.
    this Func<TMiddle, IEnumerable<TResult>> selector2,
    Func<TSource, IEnumerable<TMiddle>> selector1) =>
        value => selector1(value).SelectMany(selector2, (result1, result2) => result2);
        // Equivalent to:
        // value => selector1(value).Select(selector2).Multiply();

Or generally:

// Cannot be compiled.
public static class FuncExtensions
{
    public static Func<TSource, TMonad<TResult>> o<TMonad<>, TSource, TMiddle, TResult>( // After.
        this Func<TMiddle, TMonad<TResult>> selector2,
        Func<TSource, TMonad<TMiddle>> selector1) where TMonad<> : IMonad<TMonad<>> =>
            value => selector1(value).SelectMany(selector2, (result1, result2) => result2);
            // Equivalent to:
            // value => selector1(value).Select(selector2).Multiply();
}

Now above monad laws can be expressed by monadic selectors and Kleisli composition:

Associativity law: the Kleisli composition of monadic selectors is now the monoid multiplication, it is associative. For monadic selectors s1, s2, s3, there is (s3 ∘ s2) ∘ s1 = s3 ∘ (s2 ∘ s1).
Left unit law and right unit law: Wrap is still the monoid unit η, it is of type TSource –> TMonad<TSource>, so it can also be viewed as a monadic selector too. For monadic selector s, there is η ∘ s = s and s = s ∘ η.

internal static void KleisliComposition()
{
    Func<bool, IEnumerable<int>> selector1 =
        boolean => boolean ? new int[] { 0, 1, 2, 3, 4 } : new int[] { 5, 6, 7, 8, 9 };
    Func<int, IEnumerable<double>> selector2 = int32 => new double[] { int32 / 2D, Math.Sqrt(int32) };
    Func<double, IEnumerable<string>> selector3 =
        @double => new string[] { @double.ToString("0.0"), @double.ToString("0.00") };

    // Associativity: selector3.o(selector2).o(selector1) == selector3.o(selector2.o(selector1)).
    selector3.o(selector2).o(selector1)(true).WriteLines();
    // 0.0 0.00 0.0 0.00
    // 0.5 0.50 1.0 1.00
    // 1.0 1.00 1.4 1.41
    // 1.5 1.50 1.7 1.73
    // 2.0 2.00 2.0 2.00
    selector3.o(selector2.o(selector1))(true).WriteLines();
    // 0.0 0.00 0.0 0.00
    // 0.5 0.50 1.0 1.00
    // 1.0 1.00 1.4 1.41
    // 1.5 1.50 1.7 1.73
    // 2.0 2.00 2.0 2.00
    // Left unit: Unit.o(selector) == selector.
    Func<int, IEnumerable<int>> leftUnit = Enumerable;
    leftUnit.o(selector1)(true).WriteLines(); // 0 1 2 3 4
    selector1(true).WriteLines(); // 0 1 2 3 4
    // Right unit: selector == selector.o(Unit).
    selector1(false).WriteLines(); // 5 6 7 8 9
    Func<bool, IEnumerable<bool>> rightUnit = Enumerable;
    selector1.o(rightUnit)(false).WriteLines(); // 5 6 7 8 9
}

Kleisli category

With monad and Kleisli composition, a new kind of category called Kleisli category can be defined. Given a monad (F, ◎, η) in category C, there is a Kleisli category of F, denoted CF:

Its objects ob(CF) are ob(C), all objects in C.
Its morphisms hom(CF) are Kleisli morphisms. A Kleisli morphisms m from object X to object Y is m: X → F(Y). In DotNet, the Kleisli morphisms are above monadic selector functions.
The composition of Kleisli morphisms is the above Kleisli composition.
The identity Kleisli morphism is η of the monad, so that ηX: X → F(X).

As already demonstrated, Kleisli composition and η satisfy the category associativity law and identity law.

Monad pattern of LINQ

So LINQ SelectMany query’s quintessential mathematics is monad. Generally, in DotNet category, a type is a monad if:

This type is an open generic type definition, which can be viewed as type constructor of kind * –> *, so that it maps a concrete type to another concrete monad-wrapped type.
It is equipped with the standard LINQ query method SelectMany, which can be either instance method or extension method.
The implementation of SelectMany satisfies the monad laws, so that the monad’s monoid structure is preserved.

As Brian Beckman said in this Channel 9 video:

LINQ is monad. It is very carefully designed by Erik Meijer so that it is monad.

Eric Lippert also mentioned:

The LINQ syntax is designed specifically to make operations on the sequence monad feel natural, but in fact the implementation is more general; what C# calls "SelectMany" is a slightly modified form of the "Bind" operation on an arbitrary monad.

On the other hand, to enable the monad LINQ query expression (multiple from clauses with select clause) for a type does not require that type to be strictly a monad. This LINQ workflow syntax can be enabled for any generic or non generic type as long as it has such a SelectMany method, which can be virtually demonstrated as:

// Cannot be compiled.
internal static void Workflow<TMonad<>, T1, T2, T3, T4, TResult>( // Non generic TMonad can work too.
    Func<TMonad<T1>> operation1,
    Func<TMonad<T2>> operation2,
    Func<TMonad<T3>> operation3,
    Func<TMonad<T4>> operation4,
    Func<T1, T2, T3, T4, TResult> resultSelector) where TMonad<> : IMonad<TMonad<>>
{
    TMonad<TResult> query = from /* T1 */ value1 in /* TMonad<T1> */ operation1()
                            from /* T2 */ value2 in /* TMonad<T1> */ operation2()
                            from /* T3 */ value3 in /* TMonad<T1> */ operation3()
                            from /* T4 */ value4 in /* TMonad<T1> */ operation4()
                            select /* TResult */ resultSelector(value1, value2, value3, value4); // Define query.
}

Monad vs. monoidal/applicative functor

Monad is monoidal functor and applicative functor. Monads’ (SelectMany, Wrap) methods implement monoidal functor’s Multiply and Unit methods, and applicative functor’s (Apply, Wrap) methods. This can be virtually demonstrated as:

// Cannot be compiled.
public static partial class MonadExtensions // (SelectMany, Wrap) implements (Multiply, Unit).
{
    // Multiply: (TMonad<T1>, TMonad<T2>) => TMonad<(T1, T2)>
    public static TMonad<(T1, T2)> Multiply<TMonad<>, T1, T2>(
        this TMonad<T1> source1, TMonad<T2> source2) where TMonad<> : IMonad<TMonad<>> =>
            from value1 in source1
            from value2 in source2
            select (value1, value2);
            // source1.SelectMany(value1 => source2 (value1, value2) => value1.ValueTuple(value2));

    // Unit: Unit -> TMonad<Unit>
    public static TMonad<Unit> Unit<TMonad<>>(
        Unit unit = default) where TMonad<> : IMonad<TMonad<>> => unit.Wrap();
}

// Cannot be compiled.
public static partial class MonadExtensions // (SelectMany, Wrap) implements (Apply, Wrap).
{
    // Apply: (TMonad<TSource -> TResult>, TMonad<TSource>) -> TMonad<TResult>
    public static TMonad<TResult> Apply<TMonad<>, TSource, TResult>(
        this TMonad<Func<TSource, TResult>> selectorWrapper, 
        TMonad<TSource> source) where TMonad<> : IMonad<TMonad<>> =>
            from selector in selectorWrapper
            from value in source
            select selector(value);
            // selectorWrapper.SelectMany(selector => source, (selector, value) => selector(value));

    // Monad's Wrap is identical to applicative functor's Wrap.
}

If monad is defined with the (Multiply, Unit) methods, they implement monoidal functor’s Multiply and Unit methods, and applicative functor’s (Apply, Wrap) methods too:

// Cannot be compiled.
public static class MonadExtensions // Monad (Multiply, Unit) implements monoidal functor (Multiply, Unit).
{
    // Multiply: (TMonad<T1>, TMonad<T2>) => TMonad<(T1, T2)>
    public static TMonad<(T1, T2)> Multiply<TMonad<>, T1, T2>(
        this TMonad<T1> source1, TMonad<T2> source2) where TMonad<> : IMonad<TMonad<>> =>
            (from value1 in source1
             select (from value2 in source2
                     select (value1, value2))).Multiply();
            // source1.Select(value1 => source2.Select(value2 => (value1, value2))).Multiply();

    // Unit: Unit -> TMonad<Unit>
    public static TMonad<Unit> Unit<TMonad>(Unit unit = default) where TMonad<>: IMonad<TMonad<>> => 
        TMonad<Unit>.Unit<Unit>(unit);
}

// Cannot be compiled.
public static partial class MonadExtensions // Monad (Multiply, Unit) implements applicative functor (Apply, Wrap).
{
    // Apply: (TMonad<TSource -> TResult>, TMonad<TSource>) -> TMonad<TResult>
    public static TMonad<TResult> Apply<TMonad<>, TSource, TResult>(
        this TMonad<Func<TSource, TResult>> selectorWrapper, 
        TMonad<TSource> source)  where TMonad<> : IMonad<TMonad<>> =>
            (from selector in selectorWrapper
             select (from value in source
                     select selector(value))).Multiply();
            // selectorWrapper.Select(selector => source.Select(value => selector(value))).Multiply();

    // Wrap: TSource -> TMonad<TSource>
    public static TMonad<TSource> Wrap<TMonad<>, TSource>(
        this TSource value) where TMonad<>: IMonad<TMonad<>> => TMonad<TSource>.Unit<TSource>(value);
}

So the monad definition can be updated to implement monoidal functor and applicative functor too:

// Cannot be compiled.
public partial interface IMonad<TMonad<>> : IMonoidalFunctor<TMonad<>>, IApplicativeFunctor<TMonad<>>
{
}

More LINQ to Monads

Many other open generic type definitions provided by .NET can be monad. Take Lazy<> functor as example, first, apparently it is a type constructor of kind * –> *. Then, its SelectMany query method can be defined as extension method:

public static partial class LazyExtensions // Lazy<T> : IMonad<Lazy<>>
{
    // Multiply: Lazy<Lazy<TSource> -> Lazy<TSource>
    public static Lazy<TSource> Multiply<TSource>(this Lazy<Lazy<TSource>> sourceWrapper) =>
        sourceWrapper.SelectMany(Id, False);

    // Unit: TSource -> Lazy<TSource>
    public static Lazy<TSource> Unit<TSource>(TSource value) => Lazy(value);

    // SelectMany: (Lazy<TSource>, TSource -> Lazy<TSelector>, (TSource, TSelector) -> TResult) -> Lazy<TResult>
    public static Lazy<TResult> SelectMany<TSource, TSelector, TResult>(
        this Lazy<TSource> source,
        Func<TSource, Lazy<TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) => 
            new Lazy<TResult>(() => resultSelector(source.Value, selector(source.Value).Value));
}

Its Wrap method has been implemented previously, as a requirement of applicative functor. The following is an example of chaining operations into a workflow with Lazy<> monad:

internal static void Workflow()
{
    Lazy<string> query = from filePath in new Lazy<string>(Console.ReadLine)
                         from encodingName in new Lazy<string>(Console.ReadLine)
                         from encoding in new Lazy<Encoding>(() => Encoding.GetEncoding(encodingName))
                         from fileContent in new Lazy<string>(() => File.ReadAllText(filePath, encoding))
                         select fileContent; // Define query.
    string result = query.Value; // Execute query.
}

Since SelectMany implements deferred execution, the above LINQ query is pure and the workflow is deferred. When the query is executed by calling Lazy<>.Value, the workflow is started.

Func<> functor is also monad, with the following SelectMany:

public static partial class FuncExtensions // Func<T> : IMonad<Func<>>
{
    // Multiply: Func<Func<T> -> Func<T>
    public static Func<TSource> Multiply<TSource>(this Func<Func<TSource>> sourceWrapper) => 
        sourceWrapper.SelectMany(source => source, (source, value) => value);

    // Unit: Unit -> Func<Unit>
    public static Func<TSource> Unit<TSource>(TSource value) => Func(value);

    // SelectMany: (Func<TSource>, TSource -> Func<TSelector>, (TSource, TSelector) -> TResult) -> Func<TResult>
    public static Func<TResult> SelectMany<TSource, TSelector, TResult>(
        this Func<TSource> source,
        Func<TSource, Func<TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) => () =>
        {
            TSource value = source();
            return resultSelector(value, selector(value)());
        };
}

And the workflow is similar to Lazy<> monad’s workflow, because Lazy<T> is just a wrapper of Func<T> factory function:

internal static void Workflow()
{
    Func<string> query = from filePath in new Func<string>(Console.ReadLine)
                         from encodingName in new Func<string>(Console.ReadLine)
                         from encoding in new Func<Encoding>(() => Encoding.GetEncoding(encodingName))
                         from fileContent in new Func<string>(() => File.ReadAllText(filePath, encoding))
                         select fileContent; // Define query.
    string result = query(); // Execute query.
}

The Optional<> monad is monad too, with the following SelectMany:

public static partial class OptionalExtensions // Optional<T> : IMonad<Optional<>>
{
    // Multiply: Optional<Optional<TSource> -> Optional<TSource>
    public static Optional<TSource> Multiply<TSource>(this Optional<Optional<TSource>> sourceWrapper) =>
        sourceWrapper.SelectMany(source => source, (source, value) => value);

    // Unit: TSource -> Optional<TSource>
    public static Optional<TSource> Unit<TSource>(TSource value) => Optional(value);

    // SelectMany: (Optional<TSource>, TSource -> Optional<TSelector>, (TSource, TSelector) -> TResult) -> Optional<TResult>
    public static Optional<TResult> SelectMany<TSource, TSelector, TResult>(
        this Optional<TSource> source,
        Func<TSource, Optional<TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) => new Optional<TResult>(() =>
            {
                if (source.HasValue)
                {
                    Optional<TSelector> result = selector(source.Value);
                    if (result.HasValue)
                    {
                        return (true, resultSelector(source.Value, result.Value));
                    }
                }
                return (false, default);
            });
}

The LINQ workflow of Optional<> monad is also pure and deferred, where each operation in the chaining is an Optional<T> instance:

internal static void Workflow()
{
    string input;
    Optional<string> query =
        from filePath in new Optional<string>(() => string.IsNullOrWhiteSpace(input = Console.ReadLine())
            ? (false, default) : (true, input))
        from encodingName in new Optional<string>(() => string.IsNullOrWhiteSpace(input = Console.ReadLine())
            ? (false, default) : (true, input))
        from encoding in new Optional<Encoding>(() =>
            {
                try
                {
                    return (true, Encoding.GetEncoding(encodingName));
                }
                catch (ArgumentException)
                {
                    return (false, default);
                }
            })
        from fileContent in new Optional<string>(() => File.Exists(filePath)
            ? (true, File.ReadAllText(filePath, encoding)) : (false, default))
        select fileContent; // Define query.
    if (query.HasValue) // Execute query.
    {
        string result = query.Value;
    }
}

So Optional<> covers the scenario that each operation of the workflow may not have invalid result. When an operation has valid result (Optional<T>.HasValue returns true), its next operation executes. And when all all the operations have valid result, the entire workflow has a valid query result.

The ValueTuple<> functor is also monad. Again, its SelectMany cannot defer the call of selector, just like its Select:

public static partial class ValueTupleExtensions // ValueTuple<T, TResult> : IMonad<ValueTuple<T,>>
{
    // Multiply: ValueTuple<T, ValueTuple<T, TSource> -> ValueTuple<T, TSource>
    public static (T, TSource) Multiply<T, TSource>(this (T, (T, TSource)) sourceWrapper) =>
        sourceWrapper.SelectMany(source => source, (source, value) => value); // Immediate execution.

    // Unit: TSource -> ValueTuple<T, TSource>
    public static (T, TSource) Unit<T, TSource>(TSource value) => ValueTuple<T, TSource>(value);

    // SelectMany: (ValueTuple<T, TSource>, TSource -> ValueTuple<T, TSelector>, (TSource, TSelector) -> TResult) -> ValueTuple<T, TResult>
    public static (T, TResult) SelectMany<T, TSource, TSelector, TResult>(
        this (T, TSource) source,
        Func<TSource, (T, TSelector)> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            (source.Item1, resultSelector(source.Item2, selector(source.Item2).Item2)); // Immediate execution.
}

So its workflow is the immediate execution version of Lazy<> monad’s workflow:

public static partial class ValueTupleExtensions
{
    internal static void Workflow()
    {
        ValueTuple<string> query = from filePath in new ValueTuple<string>(Console.ReadLine())
                                   from encodingName in new ValueTuple<string>(Console.ReadLine())
                                   from encoding in new ValueTuple<Encoding>(Encoding.GetEncoding(encodingName))
                                   from fileContent in new ValueTuple<string>(File.ReadAllText(filePath, encoding))
                                   select fileContent; // Define and execute query.
        string result = query.Item1; // Query result.
    }
}

The Task<> functor is monad too. Once again, its SelectMany is immediate and impure, just like its Select:

public static partial class TaskExtensions // Task<T> : IMonad<Task<>>
{
    // Multiply: Task<Task<T> -> Task<T>
    public static Task<TResult> Multiply<TResult>(this Task<Task<TResult>> sourceWrapper) =>
        sourceWrapper.SelectMany(source => source, (source, value) => value); // Immediate execution, impure.

    // Unit: TSource -> Task<TSource>
    public static Task<TSource> Unit<TSource>(TSource value) => Task(value);

    // SelectMany: (Task<TSource>, TSource -> Task<TSelector>, (TSource, TSelector) -> TResult) -> Task<TResult>
    public static async Task<TResult> SelectMany<TSource, TSelector, TResult>(
        this Task<TSource> source,
        Func<TSource, Task<TSelector>> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            resultSelector(await source, await selector(await source)); // Immediate execution, impure.
}

So the following LINQ workflow with Task<> monad is also immediate and impure:

internal static async Task WorkflowAsync(string uri)
{
    Task<string> query = from response in new HttpClient().GetAsync(uri) // Return Task<HttpResponseMessage>.
                         from stream in response.Content.ReadAsStreamAsync() // Return Task<Stream>.
                         from text in new StreamReader(stream).ReadToEndAsync() // Return Task<string>.
                         select text; // Define and execute query.
    string result = await query; // Query result.
}

It is easy to verify all the above SelectMany methods satisfy the monad laws, and all the above (Multiply, Unit) methods preserve the monoid laws. However, not any SelectMany or (Multiply, Unit) methods can automatically satisfy those laws. Take the ValueTuple<T,> functor as example, here are its SelectMany and (Multiply, Unit):

public static partial class ValueTupleExtensions // ValueTuple<T, TResult> : IMonad<ValueTuple<T,>>
{
    // Multiply: ValueTuple<T, ValueTuple<T, TSource> -> ValueTuple<T, TSource>
    public static (T, TSource) Multiply<T, TSource>(this (T, (T, TSource)) sourceWrapper) =>
        sourceWrapper.SelectMany(source => source, (source, value) => value); // Immediate execution.

    // Unit: TSource -> ValueTuple<T, TSource>
    public static (T, TSource) Unit<T, TSource>(TSource value) => ValueTuple<T, TSource>(value);

    // SelectMany: (ValueTuple<T, TSource>, TSource -> ValueTuple<T, TSelector>, (TSource, TSelector) -> TResult) -> ValueTuple<T, TResult>
    public static (T, TResult) SelectMany<T, TSource, TSelector, TResult>(
        this (T, TSource) source,
        Func<TSource, (T, TSelector)> selector,
        Func<TSource, TSelector, TResult> resultSelector) =>
            (source.Item1, resultSelector(source.Item2, selector(source.Item2).Item2)); // Immediate execution.
}

The above (Multiply, Unit) implementations cannot preserve the monoid left unit law:

internal static void MonoidLaws()
{
    (string, int) source = ("a", 1);

    // Associativity preservation: source.Wrap().Multiply().Wrap().Multiply() == source.Wrap().Wrap().Multiply().Multiply().
    source
        .ValueTuple<string, (string, int)>()
        .Multiply()
        .ValueTuple<string, (string, int)>()
        .Multiply()
        .WriteLine(); // (, 1)
    source
        .ValueTuple<string, (string, int)>()
        .ValueTuple<string, (string, (string, int))>()
        .Multiply()
        .Multiply()
        .WriteLine(); // (, 1)
    // Left unit preservation: Unit(f).Multiply() == source.
    Unit<string, (string, int)>(source).Multiply().WriteLine(); // (, 1)
    // Right unit preservation: source == source.Select(Unit).Multiply().
    source.Select(Unit<string, int>).Multiply().WriteLine(); // (a, 1)
}

And the above SelectMany implementation breaks the left unit monad law too:

internal static void MonadLaws()
{
    ValueTuple<string, int> source = ("a", 1);
    Func<int, ValueTuple<string, char>> selector = int32 => ("b", '@');
    Func<int, ValueTuple<string, double>> selector1 = int32 => ("c", Math.Sqrt(int32));
    Func<double, ValueTuple<string, string>> selector2 = @double => ("d", @double.ToString("0.00"));
    const int Value = 5;

    // Associativity: source.SelectMany(selector1).SelectMany(selector2) == source.SelectMany(value => selector1(value).SelectMany(selector2)).
    (from value in source
        from result1 in selector1(value)
        from result2 in selector2(result1)
        select result2).WriteLine(); // (a, 1.00)
    (from value in source
        from result in (from result1 in selector1(value) from result2 in selector2(result1) select result2)
        select result).WriteLine(); // (a, 1.00)
    // Left unit: value.Wrap().SelectMany(selector) == selector(value).
    (from value in Value.ValueTuple<string, int>()
        from result in selector(value)
        select result).WriteLine(); // (, @)
    selector(Value).WriteLine(); // (b, @)
    // Right unit: source == source.SelectMany(Wrap).
    (from value in source
        from result in value.ValueTuple<string, int>()
        select result).WriteLine(); // (a, 1)
}

Category Theory via C# (6) Monoidal Functor and Applicative Functor

Wed, 25 Dec 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Category Theory via C# series]

Monoidal functor

Given monoidal categories (C, ⊗, IC) and (D, ⊛, ID), a strong lax monoidal functor is a functor F: C → D equipped with:

Monoid binary multiplication operation, which is a natural transformation φ: F(X) ⊛ F(Y) ⇒ F(X ⊗ Y)
Monoid unit, which is a morphism ι: ID → F(IC)

F preserves the monoid laws in D:

Associativity law is preserved with D’s associator αD:
Left unit law is preserved with D’s left unitor λD: and right unit law is preserved with D’s right unitor ρD:

In this tutorial, strong lax monoidal functor is called monoidal functor for short. In DotNet category, monoidal functors are monoidal endofunctors. In the definition, (C, ⊗, IC) and (D, ⊛, ID) are both (DotNet, ValueTuple<,>, Unit), so monoidal functor can be IEnumerable<T1>, IEnumerable<T2>defined as:

public interface IMonoidalFunctor<TMonoidalFunctor<>> : IFunctor<TMonoidalFunctor<>>
    where TMonoidalFunctor : IMonoidalFunctor<TMonoidalFunctor<>>
{
    // From IFunctor<TMonoidalFunctor<>>:
    // Select: (TSource -> TResult) -> (TMonoidalFunctor<TSource> -> TMonoidalFunctor<TResult>)
    // Func<TMonoidalFunctor<TSource>, TMonoidalFunctor<TResult>> Select<TSource, TResult>(Func<TSource, TResult> selector);

    // Multiply: TMonoidalFunctor<T1> x TMonoidalFunctor<T2> -> TMonoidalFunctor<T1 x T2>
    // Multiply: ValueTuple<TMonoidalFunctor<T1>, TMonoidalFunctor<T2>> -> TMonoidalFunctor<ValueTuple<T1, T2>>
    TMonoidalFunctor<ValueTuple<T1, T2>> Multiply<T1, T2>(
        ValueTuple<TMonoidalFunctor<T1>, TMonoidalFunctor<T2>> bifunctor);

    // Unit: Unit -> TMonoidalFunctor<Unit>
    TMonoidalFunctor<Unit> Unit(Unit unit);
}

Multiply accepts a ValueTuple<IEnumerable<T1>, IEnumerable<T2>> bifunctor, which is literally a 2-tuple (IEnumerable<T1>, IEnumerable<T2>). For convenience, the explicit ValueTuple<,> parameter can be represented by an implicit tuple, a pair of parameters. So the monoidal functor definition is equivalent to:

public interface IMonoidalFunctor<TMonoidalFunctor<>> : IFunctor<TMonoidalFunctor<>>
    where TMonoidalFunctor : IMonoidalFunctor<TMonoidalFunctor<>>
{
    // Multiply: TMonoidalFunctor<T1> x TMonoidalFunctor<T2> -> TMonoidalFunctor<T1 x T2>
    // Multiply: (TMonoidalFunctor<T1>, TMonoidalFunctor<T2>) -> TMonoidalFunctor<(T1, T2)>
    TMonoidalFunctor<(T1, T2)> Multiply<T1, T2>(
        TMonoidalFunctor<T1> source1, TMonoidalFunctor<T2>> source2); // Unit: Unit

    // Unit: Unit -> TMonoidalFunctor<Unit>
    TMonoidalFunctor<Unit> Unit(Unit unit);
}

IEnumerable<> monoidal functor

IEnumerable<> functor is a monoidal functor. Its Multiply method can be implemented as its extension method:

public static partial class EnumerableExtensions // IEnumerable<T> : IMonoidalFunctor<IEnumerable<>>
{
    // Multiply: IEnumerable<T1> x IEnumerable<T2> -> IEnumerable<T1 x T2>
    // Multiply: ValueTuple<IEnumerable<T1>, IEnumerable<T2>> -> IEnumerable<ValueTuple<T1, T2>>
    // Multiply: (IEnumerable<T1>, IEnumerable<T2>) -> IEnumerable<(T1, T2)>
    public static IEnumerable<(T1, T2)> Multiply<T1, T2>(
        this IEnumerable<T1> source1, IEnumerable<T2> source2) // Implicit tuple.
    {
        foreach (T1 value1 in source1)
        {
            foreach (T2 value2 in source2)
            {
                yield return (value1, value2);
            }
        }
    }

    // Unit: Unit -> IEnumerable<Unit>
    public static IEnumerable<Unit> Unit(Unit unit = default)
    {
        yield return unit;
    }
}

Now extension method Multiply can be used as a infix operator. It can be verified that the above Multiply and Unit implementations preserve the monoid laws by working with associator, left unitor and right unitor of DotNet monoidal category:

// using static Dixin.Linq.CategoryTheory.DotNetCategory;
internal static void MonoidalFunctorLaws()
{
    IEnumerable<Unit> unit = Unit();
    IEnumerable<int> source1 = new int[] { 0, 1 };
    IEnumerable<char> source2 = new char[] { '@', '#' };
    IEnumerable<bool> source3 = new bool[] { true, false };
    IEnumerable<int> source = new int[] { 0, 1, 2, 3, 4 };

    // Associativity preservation: source1.Multiply(source2).Multiply(source3).Select(Associator) == source1.Multiply(source2.Multiply(source3)).
    source1.Multiply(source2).Multiply(source3).Select(Associator).WriteLines();
        // (0, (@, True)) (0, (@, False)) (0, (#, True)) (0, (#, False))
        // (1, (@, True)) (1, (@, False)) (1, (#, True)) (1, (#, False))
    source1.Multiply(source2.Multiply(source3)).WriteLines();
        // (0, (@, True)) (0, (@, False)) (0, (#, True)) (0, (#, False))
        // (1, (@, True)) (1, (@, False)) (1, (#, True)) (1, (#, False))
    // Left unit preservation: unit.Multiply(source).Select(LeftUnitor) == source.
    unit.Multiply(source).Select(LeftUnitor).WriteLines(); // 0 1 2 3 4
    // Right unit preservation: source == source.Multiply(unit).Select(RightUnitor).
    source.Multiply(unit).Select(RightUnitor).WriteLines(); // 0 1 2 3 4
}

How could these methods be useful? Remember functor’s Select method enables selector working with value(s) wrapped by functor:

internal static void Selector1Arity(IEnumerable<int> xs)
{
    Func<int, bool> selector = x => x > 0;
    // Apply selector with xs.
    IEnumerable<bool> applyWithXs = xs.Select(selector);
}

So Select can be viewed as applying 1 arity selector (a TSource –> TResult function) with TFunctor<TSource>. For a N arity selector, to have it work with value(s) wrapped by functor, first curry it, so that it can be viewed as 1 arity function. In the following example, the (T1, T2, T3) –> TResult selector is curried to T1 –> (T2 –> T3 –> TResult) function, so that it can be viewed as only have 1 parameter, and can work with TFunctor<T1>:

internal static void SelectorNArity(IEnumerable<int> xs, IEnumerable<long> ys, IEnumerable<double> zs)
{
    Func<int, long, double, bool> selector = (x, y, z) => x + y + z > 0;

    // Curry selector.
    Func<int, Func<long, Func<double, bool>>> curriedSelector = 
        selector.Curry(); // 1 arity: x => (y => z => x + y + z > 0)
    // Partially apply selector with xs.
    IEnumerable<Func<long, Func<double, bool>>> applyWithXs = xs.Select(curriedSelector);

So partially applying the T1 –> (T2 –> T3 –> TResult) selector with TFunctor<T1> returns TFunctor<T2 –> T3 –> TResult>, where the T2 –> T3 –> TResult function is wrapped by the TFunctor<> functor. To further apply TFunctor<T2 –> T3 –> TResult> with TFunctor<T2>, Multiply can be called:

// Partially apply selector with ys.
    IEnumerable<(Func<long, Func<double, bool>>, long)> multiplyWithYs = applyWithXs.Multiply(ys);
    IEnumerable<Func<double, bool>> applyWithYs = multiplyWithYs.Select(product =>
    {
        Func<long, Func<double, bool>> partialAppliedSelector = product.Item1;
        long y = product.Item2;
        return partialAppliedSelector(y);
    });

The result of Multiply is TFunctor<(T2 –> T3 –> TResult, T2)>, where each T2 –> T3 –> TResult function is paired with each T2 value, so that each function can be applied with each value, And TFunctor<(T2 –> T3 –> TResult, T2)> is mapped to TFunctor<(T3 –> TResult)>, which can be applied with TFunctor<T3> in the same way:

// Partially apply selector with zs.
    IEnumerable<(Func<double, bool>, double)> multiplyWithZs = applyWithYs.Multiply(zs);
    IEnumerable<bool> applyWithZs = multiplyWithZs.Select(product =>
    {
        Func<double, bool> partialAppliedSelector = product.Item1;
        double z = product.Item2;
        return partialAppliedSelector(z);
    });
}

So Multiply enables applying functor-wrapped functions (TFunctor<T –> TResult>) with functor-wrapped values (TFunctor<TSource>), which returns functor-wrapped results (TFunctor<TResult>). Generally, the Multiply and Select calls can be encapsulated as the following Apply method:

// Apply: (IEnumerable<TSource -> TResult>, IEnumerable<TSource>) -> IEnumerable<TResult>
public static IEnumerable<TResult> Apply<TSource, TResult>(
    this IEnumerable<Func<TSource, TResult>> selectorWrapper, IEnumerable<TSource> source) =>
        selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2));

So that the above N arity selector application becomes:

internal static void Apply(IEnumerable<int> xs, IEnumerable<long> ys, IEnumerable<double> zs)
{
    Func<int, long, double, bool> selector = (x, y, z) => x + y + z > 0;
    // Partially apply selector with xs.
    IEnumerable<Func<long, Func<double, bool>>> applyWithXs = xs.Select(selector.Curry());
    // Partially apply selector with ys.
    IEnumerable<Func<double, bool>> applyWithYs = applyWithXs.Apply(ys);
    // Partially apply selector with zs.
    IEnumerable<bool> applyWithZs = applyWithYs.Apply(zs);
}

Applicative functor

A functor, with the above ability to apply functor-wrapped functions with functor-wrapped values, is also called applicative functor. The following is the definition of applicative functor:

// Cannot be compiled.
public interface IApplicativeFunctor<TApplicativeFunctor<>> : IFunctor<TApplicativeFunctor<>>
    where TApplicativeFunctor<> : IApplicativeFunctor<TApplicativeFunctor<>>
{
    // From: IFunctor<TApplicativeFunctor<>>:
    // Select: (TSource -> TResult) -> (TApplicativeFunctor<TSource> -> TApplicativeFunctor<TResult>)
    // Func<TApplicativeFunctor<TSource>, TApplicativeFunctor<TResult>> Select<TSource, TResult>(Func<TSource, TResult> selector);

    // Apply: (TApplicativeFunctor<TSource -> TResult>, TApplicativeFunctor<TSource> -> TApplicativeFunctor<TResult>
    TApplicativeFunctor<TResult> Apply<TSource, TResult>(
        TApplicativeFunctor<Func<TSource, TResult>> selectorWrapper, TApplicativeFunctor<TSource> source);

    // Wrap: TSource -> TApplicativeFunctor<TSource>
    TApplicativeFunctor<TSource> Wrap<TSource>(TSource value);
}

And applicative functor must satisfy the applicative laws:

Functor preservation: applying function is equivalent to applying functor-wrapped function
Identity preservation: applying functor-wrapped identity function, is equivalent to doing nothing.
Composition preservation: functor-wrapped functions can be composed by applying.
Homomorphism: applying functor-wrapped function with functor-wrapped value, is equivalent to functor-wrapping the result of applying that function with that value.
Interchange: when applying functor-wrapped functions with a functor-wrapped value, the functor-wrapped functions and the functor-wrapped value can interchange position.

IEnumerable<> applicative functor

IEnumerable<> functor is a applicative functor. Again, these methods are implemented as extension methods. And for IEnumerable<>, the Wrap method is called Enumerable to be intuitive:

public static partial class EnumerableExtensions // IEnumerable<T> : IApplicativeFunctor<IEnumerable<>>
{
    // Apply: (IEnumerable<TSource -> TResult>, IEnumerable<TSource>) -> IEnumerable<TResult>
    public static IEnumerable<TResult> Apply<TSource, TResult>(
        this IEnumerable<Func<TSource, TResult>> selectorWrapper, IEnumerable<TSource> source)
    {
        foreach (Func<TSource, TResult> selector in selectorWrapper)
        {
            foreach (TSource value in source)
            {
                yield return selector(value);
            }
        }
    }

    // Wrap: TSource -> IEnumerable<TSource>
    public static IEnumerable<TSource> Enumerable<TSource>(this TSource value)
    {
        yield return value;
    }
}

It can be verified that the above Apply and Wrap (Enumerable) implementations satisfy the applicative laws:

internal static void ApplicativeLaws()
{
    IEnumerable<int> source = new int[] { 0, 1, 2, 3, 4 };
    Func<int, double> selector = int32 => Math.Sqrt(int32);
    IEnumerable<Func<int, double>> selectorWrapper1 =
        new Func<int, double>[] { int32 => int32 / 2D, int32 => Math.Sqrt(int32) };
    IEnumerable<Func<double, string>> selectorWrapper2 =
        new Func<double, string>[] { @double => @double.ToString("0.0"), @double => @double.ToString("0.00") };
    Func<Func<double, string>, Func<Func<int, double>, Func<int, string>>> o =
        new Func<Func<double, string>, Func<int, double>, Func<int, string>>(Linq.FuncExtensions.o).Curry();
    int value = 5;

    // Functor preservation: source.Select(selector) == selector.Wrap().Apply(source).
    source.Select(selector).WriteLines(); // 0 1 1.4142135623731 1.73205080756888 2
    selector.Enumerable().Apply(source).WriteLines(); // 0 1 1.4142135623731 1.73205080756888 2
    // Identity preservation: Id.Wrap().Apply(source) == source.
    new Func<int, int>(Functions.Id).Enumerable().Apply(source).WriteLines(); // 0 1 2 3 4
    // Composition preservation: o.Wrap().Apply(selectorWrapper2).Apply(selectorWrapper1).Apply(source) == selectorWrapper2.Apply(selectorWrapper1.Apply(source)).
    o.Enumerable().Apply(selectorWrapper2).Apply(selectorWrapper1).Apply(source).WriteLines();
        // 0.0  0.5  1.0  1.5  2.0
        // 0.0  1.0  1.4  1.7  2.0 
        // 0.00 0.50 1.00 1.50 2.00
        // 0.00 1.00 1.41 1.73 2.00
    selectorWrapper2.Apply(selectorWrapper1.Apply(source)).WriteLines();
        // 0.0  0.5  1.0  1.5  2.0
        // 0.0  1.0  1.4  1.7  2.0 
        // 0.00 0.50 1.00 1.50 2.00
        // 0.00 1.00 1.41 1.73 2.00
    // Homomorphism: selector.Wrap().Apply(value.Wrap()) == selector(value).Wrap().
    selector.Enumerable().Apply(value.Enumerable()).WriteLines(); // 2.23606797749979
    selector(value).Enumerable().WriteLines(); // 2.23606797749979
    // Interchange: selectorWrapper.Apply(value.Wrap()) == (selector => selector(value)).Wrap().Apply(selectorWrapper).
    selectorWrapper1.Apply(value.Enumerable()).WriteLines(); // 2.5 2.23606797749979
    new Func<Func<int, double>, double>(function => function(value)).Enumerable().Apply(selectorWrapper1)
        .WriteLines(); // 2.5 2.23606797749979
}

Monoidal functor vs. applicative functor

The applicative functor definition is actually equivalent to above monoidal functor definition. First, applicative functor’s Apply and Wrap methods can be implemented by monoidal functor’s Multiply and Unit methods:

public static partial class EnumerableExtensions // IEnumerable<T> : IApplicativeFunctor<IEnumerable<>>
{
    // Apply: (IEnumerable<TSource -> TResult>, IEnumerable<TSource>) -> IEnumerable<TResult>
    public static IEnumerable<TResult> Apply<TSource, TResult>(
        this IEnumerable<Func<TSource, TResult>> selectorWrapper, IEnumerable<TSource> source) =>
            selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2));

    // Wrap: TSource -> IEnumerable<TSource>
    public static IEnumerable<TSource> Enumerable<TSource>(this TSource value) => Unit().Select(unit => value);
}

On the other hand, monoidal functor’s Multiply and Unit methods can be implemented by applicative functor’s Apply and Wrap methods:

public static partial class EnumerableExtensions // IEnumerable<T> : IMonoidalFunctor<IEnumerable<>>
{
    // Multiply: IEnumerable<T1> x IEnumerable<T2> -> IEnumerable<T1 x T2>
    // Multiply: (IEnumerable<T1>, IEnumerable<T2>) -> IEnumerable<(T1, T2)>
    public static IEnumerable<(T1, T2)> Multiply<T1, T2>(
        this IEnumerable<T1> source1, IEnumerable<T2> source2) =>
            new Func<T1, T2, (T1, T2)>(ValueTuple.Create).Curry().Enumerable().Apply(source1).Apply(source2);

    // Unit: Unit -> IEnumerable<Unit>
    public static IEnumerable<Unit> Unit(Unit unit = default) => unit.Enumerable();
}

Generally, for any applicative functor, its (Apply, Wrap) method pair can implement the (Multiply, Unit) method pair required as monoidal functor, and vice versa. This can be virtually demonstrated as:

// Cannot be compiled.
public static class MonoidalFunctorExtensions // (Multiply, Unit) implements (Apply, Wrap).
{
    // Apply: (TMonoidalFunctor<TSource -> TResult>, TMonoidalFunctor<TSource>) -> TMonoidalFunctor<TResult>
    public static TMonoidalFunctor<TResult> Apply<TMonoidalFunctor<>, TSource, TResult>(
        this TMonoidalFunctor<Func<TSource, TResult>> selectorWrapper, TMonoidalFunctor<TSource> source) 
        where TMonoidalFunctor<> : IMonoidalFunctor<TMonoidalFunctor<>> =>
            selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2));

    // Wrap: TSource -> TMonoidalFunctor<TSource>
    public static TMonoidalFunctor<TSource> Wrap<TMonoidalFunctor<>, TSource>(this TSource value) 
        where TMonoidalFunctor<> : IMonoidalFunctor<TMonoidalFunctor<>> =>TMonoidalFunctor<TSource>
            TMonoidalFunctor<TSource>.Unit().Select(unit => value);
}

// Cannot be compiled.
public static class ApplicativeFunctorExtensions // (Apply, Wrap) implements (Multiply, Unit).
{
    // Multiply: TApplicativeFunctor<T1> x TApplicativeFunctor<T2> -> TApplicativeFunctor<T1 x T2>
    // Multiply: (TApplicativeFunctor<T1>, TApplicativeFunctor<T2>) -> TApplicativeFunctor<(T1, T2)>
    public static TApplicativeFunctor<(T1, T2)> Multiply<TApplicativeFunctor<>, T1, T2>(
        this TApplicativeFunctor<T1> source1, TApplicativeFunctor<T2> source2) 
        where TApplicativeFunctor<> : IApplicativeFunctor<TApplicativeFunctor<>> =>
            new Func<T1, T2, (T1, T2)>(ValueTuple.Create).Curry().Wrap().Apply(source1).Apply(source2);

    // Unit: Unit -> TApplicativeFunctor<Unit>
    public static TApplicativeFunctor<Unit> Unit<TApplicativeFunctor<>>(Unit unit = default)
        where TApplicativeFunctor<> : IApplicativeFunctor<TApplicativeFunctor<>> => unit.Wrap();
}

More Monoidal functors and applicative functors

The Lazy<>, Func<>, Func<T,> functors are also monoidal/applicative functors:

public static partial class LazyExtensions // Lazy<T> : IMonoidalFunctor<Lazy<>>
{
    // Multiply: Lazy<T1> x Lazy<T2> -> Lazy<T1 x T2>
    // Multiply: (Lazy<T1>, Lazy<T2>) -> Lazy<(T1, T2)>
    public static Lazy<(T1, T2)> Multiply<T1, T2>(this Lazy<T1> source1, Lazy<T2> source2) =>
        new Lazy<(T1, T2)>(() => (source1.Value, source2.Value));

    // Unit: Unit -> Lazy<Unit>
    public static Lazy<Unit> Unit(Unit unit = default) => new Lazy<Unit>(() => unit);
}

public static partial class LazyExtensions // Lazy<T> : IApplicativeFunctor<Lazy<>>
{
    // Apply: (Lazy<TSource -> TResult>, Lazy<TSource>) -> Lazy<TResult>
    public static Lazy<TResult> Apply<TSource, TResult>(
        this Lazy<Func<TSource, TResult>> selectorWrapper, Lazy<TSource> source) =>
            selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2));

    // Wrap: TSource -> Lazy<TSource>
    public static Lazy<T> Lazy<T>(this T value) => Unit().Select(unit => value);
}

public static partial class FuncExtensions // Func<T> : IMonoidalFunctor<Func<>>
{
    // Multiply: Func<T1> x Func<T2> -> Func<T1 x T2>
    // Multiply: (Func<T1>, Func<T2>) -> Func<(T1, T2)>
    public static Func<(T1, T2)> Multiply<T1, T2>(this Func<T1> source1, Func<T2> source2) =>
        () => (source1(), source2());

    // Unit: Unit -> Func<Unit>
    public static Func<Unit> Unit(Unit unit = default) => () => unit;
}

public static partial class FuncExtensions // Func<T> : IApplicativeFunctor<Func<>>
{
    // Apply: (Func<TSource -> TResult>, Func<TSource>) -> Func<TResult>
    public static Func<TResult> Apply<TSource, TResult>(
        this Func<Func<TSource, TResult>> selectorWrapper, Func<TSource> source) =>
            selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2));

    // Wrap: TSource -> Func<TSource>
    public static Func<T> Func<T>(this T value) => Unit().Select(unit => value);
}

public static partial class FuncExtensions // Func<T, TResult> : IMonoidalFunctor<Func<T,>>
{
    // Multiply: Func<T, T1> x Func<T, T2> -> Func<T, T1 x T2>
    // Multiply: (Func<T, T1>, Func<T, T2>) -> Func<T, (T1, T2)>
    public static Func<T, (T1, T2)> Multiply<T, T1, T2>(this Func<T, T1> source1, Func<T, T2> source2) =>
        value => (source1(value), source2(value));

    // Unit: Unit -> Func<T, Unit>
    public static Func<T, Unit> Unit<T>(Unit unit = default) => _ => unit;
}

public static partial class FuncExtensions // Func<T, TResult> : IApplicativeFunctor<Func<T,>>
{
    // Apply: (Func<T, TSource -> TResult>, Func<T, TSource>) -> Func<T, TResult>
    public static Func<T, TResult> Apply<T, TSource, TResult>(
        this Func<T, Func<TSource, TResult>> selectorWrapper, Func<T, TSource> source) =>
            selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2));

    // Wrap: TSource -> Func<T, TSource>
    public static Func<T, TSource> Func<T, TSource>(this TSource value) => Unit<T>().Select(unit => value);
}

public static partial class OptionalExtensions // Optional<T> : IMonoidalFunctor<Optional<>>
{
    // Multiply: Optional<T1> x Optional<T2> -> Optional<T1 x T2>
    // Multiply: (Optional<T1>, Optional<T2>) -> Optional<(T1, T2)>
    public static Optional<(T1, T2)> Multiply<T1, T2>(this Optional<T1> source1, Optional<T2> source2) =>
        new Optional<(T1, T2)>(() => source1.HasValue && source2.HasValue
            ? (true, (source1.Value, source2.Value))
            : (false, (default, default)));

    // Unit: Unit -> Optional<Unit>
    public static Optional<Unit> Unit(Unit unit = default) =>
        new Optional<Unit>(() => (true, unit));
}

public static partial class OptionalExtensions // Optional<T> : IApplicativeFunctor<Optional<>>
{
    // Apply: (Optional<TSource -> TResult>, Optional<TSource>) -> Optional<TResult>
    public static Optional<TResult> Apply<TSource, TResult>(
        this Optional<Func<TSource, TResult>> selectorWrapper, Optional<TSource> source) =>
            selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2));

    // Wrap: TSource -> Optional<TSource>
    public static Optional<T> Optional<T>(this T value) => Unit().Select(unit => value);
}

The ValueTuple<> and Task<> functors are monoidal/applicative functors too. Notice their Multiply/Apply methods cannot defer the execution, and Task<>’s Multiply/Apply methods are impure.

public static partial class ValueTupleExtensions // ValueTuple<T> : IMonoidalFunctor<ValueTuple<>>
{
    // Multiply: ValueTuple<T1> x ValueTuple<T2> -> ValueTuple<T1 x T2>
    // Multiply: (ValueTuple<T1>, ValueTuple<T2>) -> ValueTuple<(T1, T2)>
    public static ValueTuple<(T1, T2)> Multiply<T1, T2>(this ValueTuple<T1> source1, ValueTuple<T2> source2) =>
        new ValueTuple<(T1, T2)>((source1.Item1, source2.Item1)); // Immediate execution.

    // Unit: Unit -> ValueTuple<Unit>
    public static ValueTuple<Unit> Unit(Unit unit = default) => new ValueTuple<Unit>(unit);
}

public static partial class ValueTupleExtensions // ValueTuple<T> : IApplicativeFunctor<ValueTuple<>>
{
    // Apply: (ValueTuple<TSource -> TResult>, ValueTuple<TSource>) -> ValueTuple<TResult>
    public static ValueTuple<TResult> Apply<TSource, TResult>(
        this ValueTuple<Func<TSource, TResult>> selectorWrapper, ValueTuple<TSource> source) =>
            selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2)); // Immediate execution.

    // Wrap: TSource -> ValueTuple<TSource>
    public static ValueTuple<T> ValueTuple<T>(this T value) => Unit().Select(unit => value);
}

public static partial class TaskExtensions // Task<T> : IMonoidalFunctor<Task<>>
{
    // Multiply: Task<T1> x Task<T2> -> Task<T1 x T2>
    // Multiply: (Task<T1>, Task<T2>) -> Task<(T1, T2)>
    public static async Task<(T1, T2)> Multiply<T1, T2>(this Task<T1> source1, Task<T2> source2) =>
        ((await source1), (await source2)); // Immediate execution, impure.

    // Unit: Unit -> Task<Unit>
    public static Task<Unit> Unit(Unit unit = default) => System.Threading.Tasks.Task.FromResult(unit);
}

public static partial class TaskExtensions // Task<T> : IApplicativeFunctor<Task<>>
{
    // Apply: (Task<TSource -> TResult>, Task<TSource>) -> Task<TResult>
    public static Task<TResult> Apply<TSource, TResult>(
        this Task<Func<TSource, TResult>> selectorWrapper, Task<TSource> source) =>
            selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2)); // Immediate execution, impure.

    // Wrap: TSource -> Task<TSource>
    public static Task<T> Task<T>(this T value) => Unit().Select(unit => value);
}

It is easy to verify all the above (Multiply, Unit) method pairs preserve the monoid laws, and all the above (Apply, Wrap) method pairs satisfy the applicative laws. However, not any (Multiply, Unit) or any (Apply, Wrap) can automatically satisfy the laws. Take the ValueTuple<T,> functor as example:

public static partial class ValueTupleExtensions // ValueTuple<T1, T2 : IMonoidalFunctor<ValueTuple<T,>>
{
    // Multiply: ValueTuple<T, T1> x ValueTuple<T, T2> -> ValueTuple<T, T1 x T2>
    // Multiply: (ValueTuple<T, T1>, ValueTuple<T, T2>) -> ValueTuple<T, (T1, T2)>
    public static (T, (T1, T2)) Multiply<T, T1, T2>(this (T, T1) source1, (T, T2) source2) =>
        (source1.Item1, (source1.Item2, source2.Item2)); // Immediate execution.

    // Unit: Unit -> ValueTuple<Unit>
    public static (T, Unit) Unit<T>(Unit unit = default) => (default, unit);
}

public static partial class ValueTupleExtensions // ValueTuple<T, TResult> : IApplicativeFunctor<ValueTuple<T,>>
{
    // Apply: (ValueTuple<T, TSource -> TResult>, ValueTuple<T, TSource>) -> ValueTuple<T, TResult>
    public static (T, TResult) Apply<T, TSource, TResult>(
        this (T, Func<TSource, TResult>) selectorWrapper, (T, TSource) source) =>
            selectorWrapper.Multiply(source).Select(product => product.Item1(product.Item2)); // Immediate execution.

    // Wrap: TSource -> ValueTuple<T, TSource>
    public static (T, TSource) ValueTuple<T, TSource>(this TSource value) => Unit<T>().Select(unit => value);
}

The above (Multiply, Unit) implementations cannot preserve the left unit law:

internal static void MonoidalFunctorLaws()
{
    (string, int) source = ("a", 1);
    (string, Unit) unit = Unit<string>();
    (string, int) source1 = ("b", 2);
    (string, char) source2 = ("c", '@');
    (string, bool) source3 = ("d", true);

    // Associativity preservation: source1.Multiply(source2).Multiply(source3).Select(Associator) == source1.Multiply(source2.Multiply(source3)).
    source1.Multiply(source2).Multiply(source3).Select(Associator).WriteLine(); // (b, (2, (@, True)))
    source1.Multiply(source2.Multiply(source3)).WriteLine(); // (b, (2, (@, True)))
    // Left unit preservation: unit.Multiply(source).Select(LeftUnitor) == source.
    unit.Multiply(source).Select(LeftUnitor).WriteLine(); // (, 1)
    // Right unit preservation: source == source.Multiply(unit).Select(RightUnitor).
    source.Multiply(unit).Select(RightUnitor).WriteLine(); // (a, 1)
}

And the above (Apply, Wrap) implementation breaks all applicative laws:

internal static void ApplicativeLaws()
{
    (string, int) source = ("a", 1);
    Func<int, double> selector = int32 => Math.Sqrt(int32);
    (string, Func<int, double>) selectorWrapper1 = 
        ("b", new Func<int, double>(int32 => Math.Sqrt(int32)));
    (string, Func<double, string>) selectorWrapper2 =
        ("c", new Func<double, string>(@double => @double.ToString("0.00")));
    Func<Func<double, string>, Func<Func<int, double>, Func<int, string>>> o = 
        new Func<Func<double, string>, Func<int, double>, Func<int, string>>(Linq.FuncExtensions.o).Curry();
    int value = 5;

    // Functor preservation: source.Select(selector) == selector.Wrap().Apply(source).
    source.Select(selector).WriteLine(); // (a, 1)
    selector.ValueTuple<string, Func<int, double>>().Apply(source).WriteLine(); // (, 1)
    // Identity preservation: Id.Wrap().Apply(source) == source.
    new Func<int, int>(Functions.Id).ValueTuple<string, Func<int, int>>().Apply(source).WriteLine(); // (, 1)
    // Composition preservation: o.Curry().Wrap().Apply(selectorWrapper2).Apply(selectorWrapper1).Apply(source) == selectorWrapper2.Apply(selectorWrapper1.Apply(source)).
    o.ValueTuple<string, Func<Func<double, string>, Func<Func<int, double>, Func<int, string>>>>()
        .Apply(selectorWrapper2).Apply(selectorWrapper1).Apply(source).WriteLine(); // (, 1.00)
    selectorWrapper2.Apply(selectorWrapper1.Apply(source)).WriteLine(); // (c, 1.00)
    // Homomorphism: selector.Wrap().Apply(value.Wrap()) == selector(value).Wrap().
    selector.ValueTuple<string, Func<int, double>>().Apply(value.ValueTuple<string, int>()).WriteLine(); // (, 2.23606797749979)
    selector(value).ValueTuple<string, double>().WriteLine(); // (, 2.23606797749979)
    // Interchange: selectorWrapper.Apply(value.Wrap()) == (selector => selector(value)).Wrap().Apply(selectorWrapper).
    selectorWrapper1.Apply(value.ValueTuple<string, int>()).WriteLine(); // (b, 2.23606797749979)
    new Func<Func<int, double>, double>(function => function(value))
        .ValueTuple<string, Func<Func<int, double>, double>>().Apply(selectorWrapper1).WriteLine(); // (, 2.23606797749979)
}

Category Theory via C# (5) Bifunctor

Sun, 15 Dec 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Category Theory via C# series]

Bifunctor

A functor is the mapping from 1 object to another object, with a “Select” ability to map 1 morphism to another morphism. A bifunctor (binary functor), as the name implies, is the mapping from 2 objects and from 2 morphisms. Giving category C, D and E, bifunctor F from category C, D to E is a structure-preserving morphism from C, D to E, denoted F: C × D → E:

F maps objects X ∈ ob(C), Y ∈ ob(D) to object F(X, Y) ∈ ob(E)
F also maps morphisms mC: X → X’ ∈ hom(C), mD: Y → Y’ ∈ hom(D) to morphism mE: F(X, Y) → F(X’, Y’) ∈ hom(E)

In DotNet category, bifunctors are binary endofunctors, and can be defined as:

// Cannot be compiled.
public interface IBifunctor<TBifunctor<,>> where TBifunctor<,> : IBifunctor<TBifunctor<,>>
{
    Func<TBifunctor<TSource1, TSource2>, TBifunctor<TResult1, TResult2>> Select<TSource1, TSource2, TResult1, TResult2>(
        Func<TSource1, TResult1> selector1, Func<TSource2, TResult2> selector2);
}

The most intuitive built-in bifunctor is ValueTuple<,>. Apparently ValueTuple<,> can be viewed as a type constructor of kind * –> * –> *, which accepts 2 concrete types to and return another concrete type. Its Select implementation is also straightforward:

public static partial class ValueTupleExtensions // ValueTuple<T1, T2> : IBifunctor<ValueTuple<,>>
{
    // Bifunctor Select: (TSource1 -> TResult1, TSource2 -> TResult2) -> (ValueTuple<TSource1, TSource2> -> ValueTuple<TResult1, TResult2>).
    public static Func<ValueTuple<TSource1, TSource2>, ValueTuple<TResult1, TResult2>> Select<TSource1, TSource2, TResult1, TResult2>(
        Func<TSource1, TResult1> selector1, Func<TSource2, TResult2> selector2) => source =>
            Select(source, selector1, selector2);

    // LINQ-like Select: (ValueTuple<TSource1, TSource2>, TSource1 -> TResult1, TSource2 -> TResult2) -> ValueTuple<TResult1, TResult2>).
    public static ValueTuple<TResult1, TResult2> Select<TSource1, TSource2, TResult1, TResult2>(
        this ValueTuple<TSource1, TSource2> source,
        Func<TSource1, TResult1> selector1,
        Func<TSource2, TResult2> selector2) =>
            (selector1(source.Item1), selector2(source.Item2));
}

However, similar to ValueTuple<> functor’s Select method, ValueTuple<,> bifunctor’s Select method has to call selector1 and selector2 immediately. To implement deferred execution, the following Lazy<,> bifunctor can be defined:

public class Lazy<T1, T2>
{
    private readonly Lazy<(T1, T2)> lazy;

    public Lazy(Func<(T1, T2)> factory) => this.lazy = new Lazy<(T1, T2)>(factory);

    public T1 Value1 => this.lazy.Value.Item1;

    public T2 Value2 => this.lazy.Value.Item2;

    public override string ToString() => this.lazy.Value.ToString();
}

Lazy<,> is simply the lazy version of ValueTuple<,>. Jut like Lazy<>, Lazy<,> can be constructed with a factory function, so that the call to selector1 and selector2 are deferred:

public static partial class LazyExtensions // Lazy<T1, T2> : IBifunctor<Lazy<,>>
{
    // Bifunctor Select: (TSource1 -> TResult1, TSource2 -> TResult2) -> (Lazy<TSource1, TSource2> -> Lazy<TResult1, TResult2>).
    public static Func<Lazy<TSource1, TSource2>, Lazy<TResult1, TResult2>> Select<TSource1, TSource2, TResult1, TResult2>(
        Func<TSource1, TResult1> selector1, Func<TSource2, TResult2> selector2) => source =>
            Select(source, selector1, selector2);

    // LINQ-like Select: (Lazy<TSource1, TSource2>, TSource1 -> TResult1, TSource2 -> TResult2) -> Lazy<TResult1, TResult2>).
    public static Lazy<TResult1, TResult2> Select<TSource1, TSource2, TResult1, TResult2>(
        this Lazy<TSource1, TSource2> source,
        Func<TSource1, TResult1> selector1,
        Func<TSource2, TResult2> selector2) =>
            new Lazy<TResult1, TResult2>(() => (selector1(source.Value1), selector2(source.Value2)));
}

Monoidal category

With the help of bifunctor, monoidal category can be defined. A monoidal category is a category C equipped with:

A bifunctor ⊗ as the monoid binary multiplication operation: bifunctor ⊗ maps 2 objects in C to another object in C, denoted C ⊗ C → C, which is also called the monoidal product or tensor product.
An unit object I ∈ ob(C) as the monoid unit, also called tensor unit

For (C, ⊗, I) to be a monoid, it also needs to be equipped with the following natural transformations, so that the monoid laws are satisfied:

Associator αX, Y, Z: (X ⊗ Y) ⊗ Z ⇒ X ⊗ (Y ⊗ Z) for the associativity law, where X, Y, Z ∈ ob(C)
Left unitor λX: I ⊗ X ⇒ X for the left unit law, and right unitor ρX: X ⊗ I ⇒ X for the right unit law, where X ∈ ob(C)

The following monoid triangle identity and pentagon identity diagrams still commute for monoidal category:

Here for monoidal category, the above ⊙ (general multiplication operator) becomes ⊗ (bifunctor).

Monoidal category can be simply defined as:

public interface IMonoidalCategory<TObject, TMorphism> : ICategory<TObject, TMorphism>, IMonoid<TObject> { }

DotNet category is monoidal category, with the most intuitive bifunctor ValueTuple<,> as the monoid multiplication, and Unit type as the monoid unit:

public partial class DotNetCategory : IMonoidalCategory<Type, Delegate>
{
    public static Type Multiply(Type value1, Type value2) => typeof(ValueTuple<,>).MakeGenericType(value1, value2);

    public static Type Unit => typeof(Unit);
}

To have (DotNet, ValueTuple<,>, Unit) satisfy the monoid laws, the associator, left unitor and right unitor are easy to implement:

public partial class DotNetCategory
{
    // Associator: (T1 x T2) x T3 -> T1 x (T2 x T3)
    // Associator: ValueTuple<ValueTuple<T1, T2>, T3> -> ValueTuple<T1, ValueTuple<T2, T3>>
    public static (T1, (T2, T3)) Associator<T1, T2, T3>(((T1, T2), T3) product) =>
        (product.Item1.Item1, (product.Item1.Item2, product.Item2));

    // LeftUnitor: Unit x T -> T
    // LeftUnitor: ValueTuple<Unit, T> -> T
    public static T LeftUnitor<T>((Unit, T) product) => product.Item2;

    // RightUnitor: T x Unit -> T
    // RightUnitor: ValueTuple<T, Unit> -> T
    public static T RightUnitor<T>((T, Unit) product) => product.Item1;
}

Category Theory via C# (4) Natural Transformation

Sat, 14 Dec 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Category Theory via C# series]

Natural transformation and naturality

If F: C → D and G: C → D are both functors from categories C to category D, the mapping from F to G is called natural transformation and denoted α: F ⇒ G. α: F ⇒ G is actually family of morphisms from F to G, For each object X in category C, there is a specific morphism αX: F(X) → G(X) in category D, called the component of α at X. For each morphism m: X → Y in category C and 2 functors F: C → D, G: C → D, there is a naturality square in D:

In another word, for m: X → Y in category C, there must be αY ∘ F(m) ≡ G(m) ∘ αX , or equivalently αY ∘ SelectF(m) ≡ SelectG(m) ∘ αX in category D.

In DotNet category, the following ToLazy<> generic method transforms Func<> functor to Lazy<> functor:

public static partial class NaturalTransformations
{
    // ToLazy: Func<> -> Lazy<>
    public static Lazy<T> ToLazy<T>(this Func<T> function) => new Lazy<T>(function);
}

Apparently, for above natural transformation: ToLazy<>: Func<> ⇒ Lazy<>:

for each specific object T, there is an object Func<T>, an object Lazy<T>, and a morphism ToFunc<T>: Func<T> → Lazy<T>.
For each specific morphism selector: TSource → TResult, there is a naturality square, which consists of 4 morphisms:
ToLazy<TResult>: Func<TResult> → Lazy<TResult>, which is the component of ToLazy<> at TResult
FuncExtensions.Select(selector): Func<TSource> → Func<TResult>
LazyExtensions.Select(selector): Lazy<TSource> → Lazy<TResult>
ToLazy<TSource>: Func<TSource> → Lazy<TSource>, which is the component of ToLazy<> at TSource

The following example is a simple naturality square that commutes for ToLazy<>:

internal static void Naturality()
{
    Func<int, string> selector = int32 => Math.Sqrt(int32).ToString("0.00");

    // Naturality square:
    // ToFunc<string>.o(LazyExtensions.Select(selector)) == FuncExtensions.Select(selector).o(ToFunc<int>)
    Func<Func<string>, Lazy<string>> funcStringToLazyString = ToLazy<string>;
    Func<Func<int>, Func<string>> funcInt32ToFuncString = FuncExtensions.Select(selector);
    Func<Func<int>, Lazy<string>> leftComposition = funcStringToLazyString.o(funcInt32ToFuncString);
    Func<Lazy<int>, Lazy<string>> lazyInt32ToLazyString = LazyExtensions.Select(selector);
    Func<Func<int>, Lazy<int>> funcInt32ToLazyInt32 = ToLazy<int>;
    Func<Func<int>, Lazy<string>> rightComposition = lazyInt32ToLazyString.o(funcInt32ToLazyInt32);

    Func<int> funcInt32 = () => 2;
    Lazy<string> lazyString = leftComposition(funcInt32);
    lazyString.Value.WriteLine(); // 1.41
    lazyString = rightComposition(funcInt32);
    lazyString.Value.WriteLine(); // 1.41
}

And the following are a few more examples of natural transformations:

// ToFunc: Lazy<T> -> Func<T>
public static Func<T> ToFunc<T>(this Lazy<T> lazy) => () => lazy.Value;

// ToEnumerable: Func<T> -> IEnumerable<T>
public static IEnumerable<T> ToEnumerable<T>(this Func<T> function)
{
    yield return function();
}

// ToEnumerable: Lazy<T> -> IEnumerable<T>
public static IEnumerable<T> ToEnumerable<T>(this Lazy<T> lazy)
{
    yield return lazy.Value;
}

Functor Category

Now there are functors, and mappings between functors, which are natural transformations. Naturally, they lead to category of functors. Given 2 categories C and D, there is a functor category, denoted DC:

Its objects ob(DC) are the functors from category C to D .
Its morphisms hom(DC) are the natural transformations between those functors.
The composition of natural transformations α: F ⇒ G and β: G ⇒ H, is natural transformations (β ∘ α): F ⇒ H.
The identity natural transformation idF: F ⇒ F maps each functor to itself

Regarding the category laws:

Associativity law: As fore mentioned, natural transformation’s components are morphisms in D, so natural transformation composition in DC can be viewed as morphism composition in D: (β ∘ α)X: F(X) → H(X) = (βX: G(X) → H(X)) ∘ (αX: F(X) → G(X)). Natural transformations’ composition in DC is associative, since all component morphisms’ composition in D is associative
Identity law: similarly, identity natural transform’s components are the id morphisms idF(X): F(X) → F(X) in D. Identity natural transform satisfy identity law, since all its components satisfy identity law.

Here is an example of natural transformations composition:

// ToFunc: Lazy<T> -> Func<T>
public static Func<T> ToFunc<T>(this Lazy<T> lazy) => () => lazy.Value;
#endif

// ToOptional: Func<T> -> Optional<T>
public static Optional<T> ToOptional<T>(this Func<T> function) =>
    new Optional<T>(() => (true, function()));

// ToOptional: Lazy<T> -> Optional<T>
public static Optional<T> ToOptional<T>(this Lazy<T> lazy) =>
    // new Func<Func<T>, Optional<T>>(ToOptional).o(new Func<Lazy<T>, Func<T>>(ToFunc))(lazy);
    lazy.ToFunc().ToOptional();
}

Endofunctor category

Given category C, there is a endofunctors category, denoted CC, or End(C), where the objects are the endofunctors from category C to C itself, and the morphisms are the natural transformations between those endofunctors.

All the functors in C# are endofunctors from DotNet category to DotNet. They are the objects of endofunctor category DotNetDotNet or End(DotNet).

Category Theory via C# (3) Functor and LINQ to Functors

Fri, 13 Dec 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Category Theory via C# series]

Functor and functor laws

In category theory, functor is a mapping from category to category. Giving category C and D, functor F from category C to D is a structure-preserving morphism from C to D, denoted F: C → D:

F maps objects in C to objects in D, for example, X, Y, Z, … ∈ ob(C) are mapped to F(X), F(Y), F(Z), … ∈ in ob(D)
F also maps morphisms in C to morphisms in D, for example, m: X → Y ∈ hom(C) is mapped to morphism F(m): F(X) → F(Y) ∈ hom(D). In this tutorial, to align to C#/.NET terms, this morphism mapping capability of functor is also called “select”. so F(m) is also denoted SelectF(m).

And F must satisfy the following functor laws:

Composition preservation: F(m2 ∘ m1) ≡ F(m2) ∘ F(m1), or SelectF(m2 ∘ m1) ≡ SelectF(m2) ∘ SelectF(m1), F maps composition in C to composition in D
Identity preservation: F(idX) ≡ idF(X), or SelectF(idX) ≡ idF(X), F maps each identity morphism in C to identity morphism in D

Endofunctor

When a functor F’s source category and target category are the same category C, it is called endofunctor, denoted F: C → C. In the DotNet category, there are endofunctors mapping objects (types) and morphisms (functions) in DotNet category to other objects and morphisms in itself. In C#, endofunctor in DotNet can be defined as:

// Cannot be compiled.
public interface IFunctor<TFunctor<>> where TFunctor<> : IFunctor<TFunctor>
{
    Func<TFunctor<TSource>, TFunctor<TResult>> Select<TSource, TResult>(Func<TSource, TResult> selector);
}

In DotNet category, objects are types, so the functor’s type mapping capability is represented by generic type TFunctor<>, which maps type T to another type TFunctor<T>. And in DotNet category, morphisms are functions, so the functor’s function mapping capability is represented by the Select method, which maps function of type TSource –> TResult to another function of type TFunctor<TSource> –> TFunctor<TResult>.

Unfortunately, the above interface cannot be compiled, because C#/.NET do not support higher-kinded polymorphism for types.

Type constructor and higher-kinded type

Kind is the meta type of a type:

A concrete type has the simplest kind, denoted *. All non generic types (types without type parameters) are of kind *. Closed generic types (types with concrete type arguments) are also concrete types of kind *.
An open generic type definition with type parameter can be viewed as a type constructor, which works like a function. For example, IEnumerable<> can accept a type of kind * (like int), and return another closed type of kind * (like IEnumerable<int>), so IEnumerable<> is a type constructor, its kind is denoted * –> *; ValueTuple<,> can accept 2 types of kind * (like string and bool), and return another closed type of kind * (like ValueTuple<string, bool>) so ValueTuple<,> is a type constructor, its kind is denoted (*, *) –> *, or * –> * –> * in curried style.

In above IFunctor<TFunctor<>> generic type definition, its type parameter TFunctor<> is an open generic type of kind * –> *. As a result, IFunctor<TFunctor<>> can be viewed as a type constructor, which works like a higher-order function, accepting a TFunctor<> type constructor of kind * –> *, and returning a concrete type of kind *. So IFunctor<TFunctor<>> is of kind (* –> *) –> *. This is called a higher-kinded type, and not supported by .NET and C# compiler. In another word, C# generic type definition does not support its type parameter to have type parameters. In C#, functor support is implemented by LINQ query comprehensions instead of type system.

LINQ to Functors

Built-in IEnumerable<> functor

IEnumerable<> is a built-in functor of DotNet category, which can be viewed as virtually implementing above IFunctor<TFunctor<>> interface:

public interface IEnumerable<T> : IFunctor<IEnumerable<>>, IEnumerable
{
    // Func<IEnumerable<TSource>, IEnumerable<TResult>> Select<TSource, TResult>(Func<TSource, TResult> selector);

    // Other members.
}

Endofunctor IEnumerable<> in DotNet category maps each T object (type) to IEnumerable<T> object (type), and its Select method maps TSource→ TResult morphism (function) to IEnumerable<TSource> → IEnumerable<TResult> morphism (function). So its Select method is of type (TSource –> TResult) –> (IEnumerable<TSource> –> IEnumerable<TResult>), which can be uncurried to (TSource –> TResult, IEnumerable<TSource>) –> IEnumerable<TResult>:

public interface IEnumerable<T> : IFunctor<IEnumerable<T>>, IEnumerable
{
    // Func<IEnumerable<TSource>, IEnumerable<TResult>> Select<TSource, TResult>(Func<TSource, TResult> selector);
    // can be equivalently converted to:
    // IEnumerable<TResult> Select<TSource, TResult>(Func<TSource, TResult> selector, IEnumerable<TSource> source);

    // Other members.
}

Now swap the 2 parameters of the uncurried Select, then its type becomes (IEnumerable<TSource>, TSource –> TResult) –> IEnumerable<TResult>:

public interface IEnumerable<T> : IFunctor<IEnumerable<T>>, IEnumerable
{
    // Func<IEnumerable<TSource>, IEnumerable<TResult>> Select<TSource, TResult>(Func<TSource, TResult> selector);
    // can be equivalently converted to:
    // IEnumerable<TResult> Select<TSource, TResult>(IEnumerable<TSource> source, Func<TSource, TResult> selector);

    // Other members.
}

In .NET, this equivalent version of Select is exactly the LINQ query method Select. The following is the comparison of functor Select method and LINQ Select method:

public static partial class EnumerableExtensions // IEnumerable<T> : IFunctor<IEnumerable<>>
{
    // Functor Select: (TSource -> TResult) -> (IEnumerable<TSource> -> IEnumerable<TResult>).
    public static Func<IEnumerable<TSource>, IEnumerable<TResult>> Select<TSource, TResult>(
        Func<TSource, TResult> selector) => source => 
            Select(source, selector);

    // 1. Uncurry to Select: (TSource -> TResult, IEnumerable<TSource>) -> IEnumerable<TResult>.
    // 2. Swap 2 parameters to Select: (IEnumerable<TSource>, TSource -> TResult) -> IEnumerable<TResult>.
    // 3. Define as LINQ extension method.
    public static IEnumerable<TResult> Select<TSource, TResult>(
        this IEnumerable<TSource> source, Func<TSource, TResult> selector)
    {
        foreach (TSource value in source)
        {
            yield return selector(value);
        }
    }
}

So the IEnumerable<> functor’s morphism mapping capability is implemented as the LINQ mapping query. As a part of the LINQ query expression pattern, functor support is built in in the C# language:

internal static void Map()
{
    IEnumerable<int> source = System.Linq.Enumerable.Range(0, 5);
    // Map int to string.
    Func<int, string> selector = Convert.ToString;
    // Map IEnumerable<int> to IEnumerable<string>.
    IEnumerable<string> query = from value in source
                                select selector(value); // Define query.
    query.WriteLines(); // Execute query.
}

And the above Select implementation satisfies the functor laws:

// using static Dixin.Linq.CategoryTheory.Functions;
internal static void FunctorLaws()
{
    IEnumerable<int> source = new int[] { 0, 1, 2, 3, 4 };
    Func<int, double> selector1 = int32 => Math.Sqrt(int32);
    Func<double, string> selector2 = @double => @double.ToString("0.00");

    // Associativity preservation: source.Select(selector2.o(selector1)) == source.Select(selector1).Select(selector2).
    (from value in source
        select selector2.o(selector1)(value)).WriteLines();  // 0.00 1.00 1.41 1.73 2.00
    (from value in source
        select selector1(value) into value
        select selector2(value)).WriteLines();  // 0.00 1.00 1.41 1.73 2.00
    // Identity preservation: source.Select(Id) == Id(source).
    (from value in source
        select Id(value)).WriteLines(); // 0 1 2 3 4
    Id(source).WriteLines(); // 0 1 2 3 4
}

Functor pattern of LINQ

So LINQ Select mapping query’s quintessential mathematics is functor. Generally, in DotNet category, a type is a functor if:

This type is an open generic type definition, which can be viewed as type constructor of kind * –> *, so that it maps a concrete type T to another concrete functor-wrapped type.
It is equipped with the standard LINQ query method Select, which can be either instance method or extension method.
The implementation of Select satisfies the functor laws, so that DotNet category’s associativity law and identity law are preserved.

On the other hand, to enable the LINQ functor query expression (single from clauses with select clause) for a type does not require that type to be strictly a functor. This LINQ syntax can be enabled for any generic or non generic type with as long as it has such a Select method, , which can be virtually demonstrated as:

// Cannot be compiled.
internal static void Map<TFunctor<>, TSource, TResult>( // Non generic TFunctor can work too.
    TFunctor<TSource> functor, Func<TSource, TResult> selector) where TFunctor<> : IFunctor<TFunctor<>>
{
    TFunctor<TResult> query = from /* TSource */ value in /* TFunctor<TSource> */ functor
                              select /* TResult */ selector(value); // Define query.
}

More LINQ to Functors

Many other open generic type definitions provided by .NET can be functor. Take Lazy<> as example, first, apparently it is a type constructor of kind * –> *. Then, its Select query method can be defined as extension method:

public static partial class LazyExtensions // Lazy<T> : IFunctor<Lazy<>>
{
    // Functor Select: (TSource -> TResult) -> (Lazy<TSource> -> Lazy<TResult>)
    public static Func<Lazy<TSource>, Lazy<TResult>> Select<TSource, TResult>(
        Func<TSource, TResult> selector) => source =>
            Select(source, selector);

    // LINQ Select: (Lazy<TSource>, TSource -> TResult) -> Lazy<TResult>
    public static Lazy<TResult> Select<TSource, TResult>(
        this Lazy<TSource> source, Func<TSource, TResult> selector) =>
            new Lazy<TResult>(() => selector(source.Value));

    internal static void Map()
    {
        Lazy<int> source = new Lazy<int>(() => 1);
        // Map int to string.
        Func<int, string> selector = Convert.ToString;
        // Map Lazy<int> to Lazy<string>.
        Lazy<string> query = from value in source
                             select selector(value); // Define query.
        string result = query.Value; // Execute query.
    }
}

Func<> with 1 type parameter is also a functor with the following Select implementation:

public static partial class FuncExtensions // Func<T> : IFunctor<Func<>>
{
    // Functor Select: (TSource -> TResult) -> (Func<TSource> -> Func<TResult>)
    public static Func<Func<TSource>, Func<TResult>> Select<TSource, TResult>(
        Func<TSource, TResult> selector) => source =>
            Select(source, selector);

    // LINQ Select: (Func<TSource>, TSource -> TResult) -> Func<TResult>
    public static Func<TResult> Select<TSource, TResult>(
        this Func<TSource> source, Func<TSource, TResult> selector) =>
            () => selector(source());

    internal static void Map()
    {
        Func<int> source = () => 1;
        // Map int to string.
        Func<int, string> selector = Convert.ToString;
        // Map Func<int> to Func<string>.
        Func<string> query = from value in source
                             select selector(value); // Define query.
        string result = query(); // Execute query.
    }
}

Here Select maps TSource –> TResult function to Func<TSource> –> Func<TResult> function, which is straightforward. The other Func generic delegate types, like Func<,> with 2 type parameters, could be more interesting. Just like fore mentioned ValueTuple<,>, Func<,> is of kind * –> * –> *, and can be viewed as a type constructor accepting 2 concrete types and returning another concrete type, which is different from functor. However, if Func<,> already has a concrete type T as its first type parameter, then Func<T,> can be viewed as a partially applied type constructor of kind * –> *, which can map one concrete type (its second type parameter) to another concrete type. So that Func<T,> is also a functor, with the following Select method:

public static partial class FuncExtensions // Func<T, TResult> : IFunctor<Func<T,>>
{
    // Functor Select: (TSource -> TResult) -> (Func<T, TSource> -> Func<T, TResult>)
    public static Func<Func<T, TSource>, Func<T, TResult>> Select<T, TSource, TResult>(
        Func<TSource, TResult> selector) => source =>
            Select(source, selector);

    // LINQ Select: (Func<T, TSource>, TSource -> TResult) -> Func<T, TResult>
    public static Func<T, TResult> Select<T, TSource, TResult>(
        this Func<T, TSource> source, Func<TSource, TResult> selector) =>
            value => selector(source(value)); // selector.o(source);
}

This time Select maps TSource –> TResult function to Func<T, TSource> –> Func<T, TResult> function. Actually, Func<T,> functor’s Select is exactly the function composition:

internal static void Map<T>(T input)
{
    Func<T, string> source = value => value.ToString();
    // Map string to bool.
    Func<string, bool> selector = string.IsNullOrWhiteSpace;
    // Map Func<T, string> to Func<T, bool>.
    Func<T, bool> query = from value in source
                          select selector(value); // Define query.
    bool result = query(input); // Execute query.

    // Equivalent to:
    Func<T, string> function1 = source;
    Func<string, bool> function2 = selector;
    Func<T, bool> composition = function2.o(function1);
    result = composition(input);
}

ValueTuple<> with 1 type parameter simply wraps a value. It is the eager version of Lazy<>, and it is also functor, with the following Select method:

public static partial class ValueTupleExtensions // ValueTuple<T> : IFunctor<ValueTuple<>>
{
    // Functor Select: (TSource -> TResult) -> (ValueTuple<TSource> -> ValueTuple<TResult>)
    public static Func<ValueTuple<TSource>, ValueTuple<TResult>> Select<TSource, TResult>(
        Func<TSource, TResult> selector) => source =>
            Select(source, selector); // Immediate execution.

    // LINQ Select: (ValueTuple<TSource>, TSource -> TResult) -> ValueTuple<TResult>
    public static ValueTuple<TResult> Select<TSource, TResult>(
        this ValueTuple<TSource> source, Func<TSource, TResult> selector) =>
            new ValueTuple<TResult>(selector(source.Item1)); // Immediate execution.
}

Unlike all the previous Select, here ValueTuple<>’s Select query method cannot implement deferred execution. To construct a ValueTuple<TResult> instance and return, selector must be called immediately to evaluate the result value.

internal static void Map()
{
    ValueTuple<int> source = new ValueTuple<int>(1);
    // Map int to string.
    Func<int, string> selector = int32 =>
        {
            $"{nameof(selector)} is called with {int32}.".WriteLine();
            return Convert.ToString(int32);
        };
    // Map ValueTuple<int> to ValueTuple<string>.
    ValueTuple<string> query = from value in source // Define and execute query.
                                select selector(value); // selector is called with 1.
    string result = query.Item1; // Query result.
}

Similar to Func<T,>, ValueTuple<T,> is also functor, with the following Select method of immediate execution:

public static partial class ValueTupleExtensions // ValueTuple<T, T2> : IFunctor<ValueTuple<T,>>
{
    // Functor Select: (TSource -> TResult) -> (ValueTuple<T, TSource> -> ValueTuple<T, TResult>)
    public static Func<(T, TSource), (T, TResult)> Select<T, TSource, TResult>(
        Func<TSource, TResult> selector) => source =>
            Select(source, selector); // Immediate execution.

    // LINQ Select: (ValueTuple<T, TSource>, TSource -> TResult) -> ValueTuple<T, TResult>
    public static (T, TResult) Select<T, TSource, TResult>(
        this(T, TSource) source, Func<TSource, TResult> selector) =>
            (source.Item1, selector(source.Item2)); // Immediate execution.

    internal static void Map<T>(T item1)
    {
        (T, int) source = (item1, 1);
        // Map int to string.
        Func<int, string> selector = int32 =>
        {
            $"{nameof(selector)} is called with {int32}.".WriteLine();
            return Convert.ToString(int32);
        };
        // Map ValueTuple<T, int> to ValueTuple<T, string>.
        (T, string) query = from value in source // Define and execute query.
                            select selector(value); // selector is called with 1.
        string result = query.Item2; // Query result.
    }
}

Task is also an example of functor, with the following Select method:

public static partial class TaskExtensions // Task<T> : IFunctor<Task<>>
{
    // Functor Select: (TSource -> TResult) -> (Task<TSource> -> Task<TResult>)
    public static Func<Task<TSource>, Task<TResult>> Select<TSource, TResult>(
        Func<TSource, TResult> selector) => source =>
            Select(source, selector); // Immediate execution, impure.

    // LINQ Select: (Task<TSource>, TSource -> TResult) -> Task<TResult>
    public static async Task<TResult> Select<TSource, TResult>(
        this Task<TSource> source, Func<TSource, TResult> selector) =>
            selector(await source); // Immediate execution, impure.

    internal static async Task MapAsync()
    {
        Task<int> source = System.Threading.Tasks.Task.FromResult(1);
        // Map int to string.
        Func<int, string> selector = Convert.ToString;
        // Map Task<int> to Task<string>.
        Task<string> query = from value in source
                             select selector(value); // Define and execute query.
        string result = await query; // Query result.
    }
}

Similar to ValueTuple<>, above Select implementation is not deferred either. When Select is called, if the source task is already completed, the selector function is called immediately. And unlike all the previous Select methods are pure (referential transparent and side effect free), this Select use the await syntactic sugar to construct a state machine and start it immediately. So it changes state and is impure.

Nullable<> is also an interesting type. It is of kind * –> * and the following Select method can be defined:

public static partial class NullableExtensions // Nullable<T> : IFunctor<Nullable<>>
{
    // Functor Select: (TSource -> TResult) -> (Nullable<TSource> -> Nullable<TResult>)
    public static Func<TSource?, TResult?> Select2<TSource, TResult>(
        Func<TSource, TResult> selector) where TSource : struct where TResult : struct => source =>
            Select(source, selector); // Immediate execution.

    // LINQ Select: (Nullable<TSource>, TSource -> TResult) -> Nullable<TResult>
    public static TResult? Select<TSource, TResult>(
        this TSource? source, Func<TSource, TResult> selector) where TSource : struct where TResult : struct =>
            source.HasValue ? selector(source.Value) : default; // Immediate execution.

    internal static void Map()
    {
        long? source1 = 1L;
        // Map int to string.
        Func<long, TimeSpan> selector = TimeSpan.FromTicks;
        // Map Nullable<int> to Nullable<TimeSpan>.
        TimeSpan? query1 = from value in source1
                           select selector(value); // Define and execute query.
        TimeSpan result1 = query1.Value; // Query result.

        long? source2 = null;
        // Map Nullable<int> to Nullable<TimeSpan>.
        TimeSpan? query2 = from value in source2
                           select selector(value); // Define and execute query.
        bool result2 = query2.HasValue; // Query result.
    }
}

In the above Select method, if the source Nullable<TSource> instance represents an actual value of TSource, that value is extracted to call selector, and the result is wrapped into another Nullable<TResult> instance to return; if the source represents null, selector is not called, and a Nullable<TResult> instance representing null is directly returned. There are 2 issues here. First, Nullable<>’s type parameter is constrained to be structures, so it can only map some objects of DotNet category (the value types). Second, the Select implementation cannot be deferred. As LINQ query method, deferred execution is always preferred whenever possible. So the following Optional<T> type can be defined to be used with any type parameter, and also be lazy:

public readonly struct Optional<T>
{
    private readonly Lazy<(bool, T)> factory;

    public Optional(Func<(bool, T)> factory = null) =>
        this.factory = factory == null ? null : new Lazy<(bool, T)>(factory);

    public bool HasValue => this.factory?.Value.Item1 ?? false;

    public T Value
    {
        get
        {
            if (!this.HasValue)
            {
                throw new InvalidOperationException($"{nameof(Optional<T>)} object must have a value.");
            }
            return this.factory.Value.Item2;
        }
    }
}

Optional<T> is still a structure just like Nullable<T>, so its instance cannot be null. Its parameter is not constrained, so it can wrap any valid or invalid value of any type, Its constructor accepts a factory function just like Lazy<>, s the evaluation of its wrapped value can be deferred. And the factory function returns a tuple of bool value and T value, where the bool value indicates if the other T value is a valid value, and that bool value can be returned by the the HasValue property.

internal static void Optional()
{
    int int32 = 1;
    Func<int, string> function = Convert.ToString;

    Nullable<int> nullableInt32 = new Nullable<int>(int32);
    Nullable<Func<int, string>> nullableFunction = new Nullable<Func<int, string>>(function); // Cannot be compiled.
    Nullable<string> nullableString = new Nullable<string>(); // Cannot be compiled.

    Optional<int> optionalInt32 = new Optional<int>(() => (true, int32));
    Optional<Func<int, string>> optionalFunction = new Optional<Func<int, string>>(() => true, function));
    Optional<string> optionalString = new Optional<string>(); // Equivalent to: new Optional<string>(() => false, default);
}

Apparently, Optional<> is a factor, and its Select can be defined with deferred execution:

public static partial class OptionalExtensions // Optional<T> : IFunctor<Optional<>>
{
    // Functor Select: (TSource -> TResult) -> (Optional<TSource> -> Optional<TResult>)
    public static Func<Optional<TSource>, Optional<TResult>> Select<TSource, TResult>(
        Func<TSource, TResult> selector) => source =>
            Select(source, selector);

    // LINQ Select: (Optional<TSource>, TSource -> TResult) -> Optional<TResult>
    public static Optional<TResult> Select<TSource, TResult>(
        this Optional<TSource> source, Func<TSource, TResult> selector) =>
            new Optional<TResult>(() => source.HasValue
                ? (true, selector(source.Value)) : (false, default));

    internal static void Map()
    {
        Optional<int> source1 = new Optional<int>(() => (true, 1));
        // Map int to string.
        Func<int, string> selector = Convert.ToString;
        // Map Optional<int> to Optional<string>.
        Optional<string> query1 = from value in source1
                                    select selector(value); // Define query.
        if (query1.HasValue) // Execute query.
        {
            string result1 = query1.Value;
        }

        Optional<int> source2 = new Optional<int>();
        // Map Optional<int> to Optional<string>.
        Optional<string> query2 = from value in source2
                                    select selector(value); // Define query.
        if (query2.HasValue) // Execute query.
        {
            string result2 = query2.Value;
        }
    }
}

It is easy to verify all the above Select methods satisfy the functor laws. However, not any Select can automatically satisfy the functor laws. The following is a different Select implementation for Lazy<>:

public static Lazy<TResult> Select<TSource, TResult>(
    this Lazy<TSource> source, Func<TSource, TResult> selector) =>
        new Lazy<TResult>(() => default);

And it breaks the functor because it does not preserve the identity law:

internal static void FunctorLaws()
{
    Lazy<int> lazy = new Lazy<int>(() => 1);
    Func<int, string> selector1 = Convert.ToString;
    Func<string, double> selector2 = Convert.ToDouble;

    // Associativity preservation: TFunctor<T>.Select(f2.o(f1)) == TFunctor<T>.Select(f1).Select(f2)
    lazy.Select(selector2.o(selector1)).Value.WriteLine(); // 0
    lazy.Select(selector1).Select(selector2).Value.WriteLine(); // 0
    // Identity preservation: TFunctor<T>.Select(Id) == Id(TFunctor<T>)
    lazy.Select(Id).Value.WriteLine(); // 0
    Id(lazy).Value.WriteLine(); // 1
}

Category Theory via C# (2) Monoid

Thu, 12 Dec 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Category Theory via C# series]

Monoid and monoid laws

Monoid is an important algebraic structure in category theory. A monoid M is a set M equipped with a binary operation ⊙ and a special element I, denoted 3-tuple (M, ⊙, I), where

M is a set of elements
⊙ is a binary operator called multiplication, so that M ⊙ M → M, which means the multiplication of 2 elements in set M always results an element in set M. This operation is also denoted μ, so that μ(M, M) ≡ M ⊙ M.
I is a special unit element (aka neutral element, or identity element) in set M

And the binary operator and the unit element must satisfy the following monoid laws:

associative law αX, Y, Z: (X ⊙ Y) ⊙ Z ≡ X ⊙ (Y ⊙ Z), where X ∈ M, Y ∈ M, Z ∈ M
left unit law λX: I ⊙ X ≡ X, where X ∈ M; and right unit law ρX: X ≡ X ⊙ I, where X ∈ M

so that:

the triangle identity commutes:
and the pentagon identity commutes::

In C#, the monoid definition can be represented as:

public interface IMonoid<T>
{
    static abstract T Multiply(T value1, T value2);

    static abstract T Unit { get; }
}

An intuitive example is the set ℤ of all integers, with binary operator + and unit element 0. The addition of 2 integers always results another integer; Also for integers x, y ,z, there is (x + y) + z ≡ x + (y + z) and 0 + x ≡ x ≡ x + 0 (ℤ, +, 0), so that (ℤ, +, 0) is monoid. Apparently (ℤ, *, 1) is monoid too.

public class Int32SumMonoid : IMonoid<int>
{
    public static int Multiply(int value1, int value2) => value1 + value2;

    public static int Unit => GenericIndirection.AdditiveUnit<int>(); // 0.
}

public class Int32ProductMonoid : IMonoid<int>
{
    public static int Multiply(int value1, int value2) => value1 * value2;

    public static int Unit => GenericIndirection.MultiplicativeUnit<int>(); // 1.
}

public static class GenericIndirection
{
    public static T AdditiveUnit<T>() where T : IAdditiveIdentity<T, T> => T.AdditiveIdentity;

    public static T MultiplicativeUnit<T>() where T : IMultiplicativeIdentity<T, T> => T.MultiplicativeIdentity;
}

Another monoid example is (string, string.Concat, string.Empty):

public class StringConcatMonoid : IMonoid<string>
{
    public static string Multiply(string value1, string value2) => string.Concat(value1, value2);

    public static string Unit => string.Empty;
}

Here string can be viewed as a sequence of char, and the unit string is an empty sequence. Generally, (IEnumerable<T>, Enumerable.Concat<T>, Enumerable.Empty<T>()) is a monoid:

public class EnumerableConcatMonoid<T> : IMonoid<IEnumerable<T>>
{
    public static IEnumerable<T> Multiply(IEnumerable<T> value1, IEnumerable<T> value2) => value1.Concat(value2);

    public static IEnumerable<T> Unit => Enumerable.Empty<T>();
}

The set of Boolean values { true, false }, with binary operator && and unit element true, is a monoid with only 2 element: ({true, false}, &&, true); And so is ({ true, false }, ||, false):

public class BooleanAndMonoid : IMonoid<bool>
{
    public static bool Multiply(bool value1, bool value2) => value1 && value2;

    public static bool Unit => true;
}

public class BooleanOrMonoid : IMonoid<bool>
{
    public static bool Multiply(bool value1, bool value2) => value1 || value2;

    public static bool Unit => false;
}

A monoid with 1 single element can be defined with the special type void (System.Void):

namespace System
{
    [ComVisible(true)]
    [Serializable]
    [StructLayout(LayoutKind.Sequential, Size = 1)]
    public struct Void
    {
    }
}

The set { default(void) } with the following operator and unit is a monoid:

public class VoidMonoid : IMonoid<void>
{
    public void Multiply(void value1, void value2) => default;

    public void Unit() => default;
}

However, C# compiler does not allow void keyword or System.Void type to be used in this way, so here System.Void can be replaced with its equivalency in F#, the Microsoft.FSharp.Core.Unit type:

namespace Microsoft.FSharp.Core
{
    [CompilationMapping(SourceConstructFlags.ObjectType)]
    [Serializable]
    public sealed class Unit : IComparable
    {
        internal Unit() { }

        public override int GetHashCode() => 0;

        public override bool Equals(object obj) => 
            obj == null || LanguagePrimitives.IntrinsicFunctions.TypeTestGeneric<Unit>(obj);

        int IComparable.CompareTo(object obj) => 0;
    }
}

With an internal constructor, Unit cannot be instantiated, a Unit variable can only be default(Unit), which is null. So similarly, set { default(Unit) } with the following operator and unit element is monoid:

public class UnitMonoid : IMonoid<Unit>
{
    public static Unit Multiply(Unit value1, Unit value2) => default;

    public static Unit Unit => default;
}

Monoid as category

An individual monoid (M, ⊙, I) can be a category C with 1 single object M, where:

The collection of objects ob(C) is { M }, which means, category C has 1 single object, that is M.
The collection of orphisms hom(C) is set M itself, which mean, each elements in set M is a morphism in category C.
The composition operation ∘ of C is ⊙: since each morphism in C is each element in M, the composition of morphisms is just the multiplication of elements.
The identity morphism id of C is unit element I: the identity morphism in C is the unit element in M

In this way, since M, ⊙, I satisfies the monoid laws, apparently the category laws are satisfied. In C#, this singleton category can be represented as:

public class MonoidCategory<T, TMonoid> : ICategory<Type, T> where TMonoid : IMonoid<T>
{
    public static IEnumerable<Type> Objects { get { yield return typeof(TMonoid); } }

    public static T Compose(T morphism2, T morphism1) => TMonoid.Multiply(morphism1, morphism2);

    public static T Id(Type @object) => TMonoid.Unit;
}

Category Theory via C# (1) Fundamentals

Sun, 01 Dec 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Category Theory via C# series]

Category theory is a theoretical framework to describe abstract structures and relations in mathematics, first introduced by Samuel Eilenberg and Saunders Mac Lane in 1940s. It examines mathematical concepts and properties in an abstract way, by formalizing them as collections of items and their relations. Category theory is abstract, and called "general abstract nonsense" by Norman Steenrod; It is also general, therefore widely applied in many areas in mathematics, physics, and computer science, etc. For programming, category theory is the algebraic theory of types and functions, and also the rationale and foundation of LINQ and any functional programming. This chapter discusses category theory and its important concepts, including category, morphism, natural transform, monoid, functor, and monad, etc. These general abstract concepts will be demonstrated with intuitive diagrams and specific C# and LINQ examples. These knowledge also helps building a deep understanding of functional programming in C# or other languages, since any language with types and functions is a category-theoretic structure.

Category and category laws

In category theory, a category C is a algebraic structure consists of the following 3 kinds of mathematical entities:

A collection of objects, denoted ob(C). This is not the objects in object-oriented programming paradigm.
A collection of morphisms (relations, aka arrows or maps) between objects, denoted hom(C). A morphism m from source object X to target object Y is denoted m: X → Y.
A composition operation of morphisms, denoted ∘. For m1: X → Y and m2: Y → Z, their composition is also a morphism (m2∘ m1): Y → Z. Here the name of m1 of m2 also implies the order. m2 ∘ m1 can be read as m2 after m1.

And these entities must satisfy the following 2 category laws:

Associative law: the composition of morphisms associative: For m1: W → X, m2: X → Y and m3: Y → Z, there is (m3 ∘ m2) ∘ m1≡ ≡ m3 ∘ (m2 ∘ m1).
Identity law: for each object X, there is an identity morphism: idx : X → X, and identity morphism is neutral for morphism composition. For m: X → Y, there is idY ∘ m ≡ m ≡ m ∘ idX.

To make above abstract definitions intuitive, a category can be represented by the following interface:

public interface ICategory<TObject, TMorphism>
{
    static abstract IEnumerable<TObject> Objects { get; }

    static abstract TMorphism Compose(TMorphism morphism2, TMorphism morphism1);

    static abstract TMorphism Id(TObject @object);
}

A simple example of category is the category of integers, where the collection of objects are all integers, and the collection of morphisms are ≤ (less than or equal to) relations, from an integer either to itself, or to another integer greater than or equal to it, for example: m1: 0 → 1 (0 ≤ 1), m2: 1 → 10 (1 ≤ 10), etc. Regarding the transitivity of inequality, the ≤ morphisms can be composed, for example, m1: 0 → 1 (0 ≤ 1) and m2: 1 → 10 (1 ≤ 10) can be composed to another morphism (m2 ∘ m1): 0 → 10 (0 ≤ 10).

Apparently, the above composition is associative, foe example: ((1 ≤ 10) ∘ (0 ≤ 1)) ∘ (-1 ≤ 0) ≡ -1 ≤ 10 ≡ (1 ≤ 10) ∘ ((0 ≤ 1) ∘ (-1 ≤ 0)). And for each integer X, there is an identity morphism idX: X → X (X ≤ X), and (Y ≤ Y) ∘ (X ≤ Y) ≡ X ≤ Y ≡ (X ≤ Y) ∘ (X ≤ X). So the category laws are satisfied. In C#, integer can be represented by int, and the morphism of ≤ relation can be represented by a BinaryExpression of node type LessThanOrEqual, so the category can be represented as:

public class Int32Category : ICategory<int, BinaryExpression>
{
    public static IEnumerable<int> Objects
    {
        get
        {
            for (int int32 = int.MinValue; int32 <= int.MaxValue; int32++)
            {
                yield return int32;
            }
        }
    }

    public static BinaryExpression Compose(BinaryExpression morphism2, BinaryExpression morphism1) =>
        Expression.LessThanOrEqual(morphism2.Left, morphism1.Right); // (Y <= Z) ∘ (X <= Y) => X <= Z.

    public static BinaryExpression Id(int @object) =>
        Expression.GreaterThanOrEqual(Expression.Constant(@object), Expression.Constant(@object)); // X <= X.
}

DotNet category

.NET can also be viewed as a category of types and functions, called DotNet:

ob(DotNet): the collection of objects in DotNet category are .NET types, like string (System.String), int (System.Int32), bool (System.Boolean), etc.
hom(DotNet): the collection of morphisms in DotNet category are .NET pure functions between the input type (source object) to the output type (target object), like int.Parse: string → int, DateTime.IsLeapYear: int → bool, etc.
∘: in DotNet category, the composition operation of morphisms is the composition of functions.

As already discussed in lambda calculus chapter, function composition is associative, and the unit function Id is the identity morphism:

public static partial class Functions
{
    public static Func<TSource, TResult> o<TSource, TMiddle, TResult>(
        this Func<TMiddle, TResult> function2, Func<TSource, TMiddle> function1) =>
            value => function2(function1(value));

    public static TSource Id<TSource>(T value) => value;
}

So that the category laws are satisfied.

The DotNet category can be represented as:

public partial class DotNetCategory : ICategory<Type, Delegate>
{
    public static IEnumerable<Type> Objects => AppDomain.CurrentDomain.GetAssemblies()
        .SelectMany(assembly => assembly.ExportedTypes);

    public static Delegate Compose(Delegate morphism2, Delegate morphism1) =>
        // return (Func<TSource, TResult>)Functions.Compose<TSource, TMiddle, TResult>(
        //    (Func<TMiddle, TResult>)morphism2, (Func<TSource, TMiddle>)morphism1);
        (Delegate)typeof(Tutorial.FuncExtensions).GetMethod(nameof(Tutorial.FuncExtensions.o))
            .MakeGenericMethod( // TSource, TMiddle, TResult.
                morphism1.Method.GetParameters().Single().ParameterType,
                morphism1.Method.ReturnType,
                morphism2.Method.ReturnType)
            .Invoke(null, new object[] { morphism2, morphism1 });

    public static Delegate Id(Type @object) => // Functions.Id<TSource>
        typeof(Functions).GetMethod(nameof(Functions.Id)).MakeGenericMethod(@object)
            .CreateDelegate(typeof(Func<,>).MakeGenericType(@object, @object));
}

In DotNet category, each object is a type represented by System.Type, so Objects method queries all available types in current assembly, and also recursively query all available assemblies in all reference assemblies. And each morphism is a function from one type to another, which can be represented by System.Delegate, so the composition is just to call the o operator with 2 Delegate instances.

Lambda Calculus via C# (8) Undecidability of Equivalence

Thu, 28 Nov 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Lambda Calculus via C# series]

All the previous parts demonstrated what lambda calculus can do – defining functions to model the computing, applying functions to execute the computing, implementing recursion, encoding data types and data structures, etc. Lambda calculus is a powerful tool, and it is Turing complete. This part discuss some interesting problem that cannot be done with lambda calculus – asserting whether 2 lambda expressions are equivalent.

Assuming f1 and f2 are 2 functions, they are equivalent if for ∀x, there is f1 x ≡ f2 x. For example, the following 2 functions can alpha-convert to each other:

f1 := λx.Add x 1
f2 := λy.Add y 1

Apparently they are equivalent. And they are both equivalent to:

f3 := λx.Add 1 x

because Add is commutative. Undecidability of equivalence means, in lambda calculus, there is no function can takes 2 lambda expressions as input, and returns True/False to indicate whether those 2 lambda expressions are equivalent or not. Alonzo Church has a proof using normal form. An intuitive proof can be done by viewing equivalence problem as another version of halting problem. Actually, Alonzo Church’s publish on equivalence is earlier (April 1936) than Alan Turing’s publish on halting problem (May 1936). To make it simple, this part discusses the undecidability of halting problem first, then discuss the undecidability of equivalence.

Halting problem

The halting problem is the problem of determining, when running an arbitrary program with an input, whether the program halts (finish running) or does not halt (run forever). For example:

Function Increase halts (finish running) with argument x, and returns x + 1.
Function ω does not halt with argument ω, Ω := ω ω reduces (runs) forever.

No general algorithm can solve the halting problem for all possible program-input pairs. To prove this, first define a simple function Sequence.

Sequence := λa.λb.b

When applying Sequence, the reduction strategy matters. In normal order, both its first argument is never reduced. In this part, applicative order is always assumed - the same reduction strategy as C#. So Sequence can be viewed as - reduce (run) a then reduce (run) b sequentially, and return the reduction result of b. When applying Sequence with Ω and another lambda expression. It reduces forever in applicative order:

Sequence Ω x
≡ Sequence (ω ω) x
≡ Sequence ((λx.x x) (λx.x x)) x
≡ Sequence ((λx.x x) (λx.x x)) x
≡ ...

Because Ω does not halt, Sequence Ω does not halt either. In C#:

public static partial class Functions<T1, T2>
{
    public static readonly Func<T1, Func<T2, T2>> 
        Sequence = value1 => value2 => value2;
}

Assume an IsHalting function exists, which takes 2 parameters f and x, and returns True/False if function f halts/does not halt with parameter x:

IsHalting := λf.λx.If (/* f halts with x */) (λx.True) (λx.False)

Then an IsNotHalting function can be defined to test whether function f does not halt with argument f (itself):

IsNotHalting := λf.If (IsHalting f f) (λx.Sequence Ω False) (λx.True)

When a certain function f does not halt with itself, by definition IsNotHalting f returns True:

IsNotHalting f
≡ If (IsHalting f f) (λx.Sequence Ω False) (λx.True))
≡ If (False) (λx.Sequence Ω False) (λx.True))
≡ True

Remember the If function is lazy, here λx.Sequence Ω False is never reduced. When f halts with itself, the application reduces to Sequence Ω False:

IsNotHalting f
≡ If (IsHalting f f) (λx.Sequence Ω False) (λx.True))
≡ If (True) (λx.Sequence Ω False) (λx.True))
≡ Sequence Ω False
≡ Sequence (ω ω) False
≡ Sequence ((λx.x x) (λx.x x)) False
≡ Sequence ((λx.x x) (λx.x x)) False
≡ ...

As fore mentioned, Sequence Ω does not halt. So in this case, IsNotHalting f never returns False.

In C# IsHalting and IsNotHalting functions can be represented as:

internal static class Halting<T, TResult>
{
    // IsHalting = f => x => True if f halts with x; otherwise, False
    internal static readonly Func<Func<T, TResult>, Func<T, Boolean>>
        IsHalting = f => x => throw new NotImplementedException();

    // IsNotHalting = f => If(IsHalting(f)(f))(_ => Sequence(Ω)(False))(_ => True)
    internal static readonly Func<SelfApplicableFunc<TResult>, Boolean>
        IsNotHalting = f =>
            If(Halting<SelfApplicableFunc<TResult>, TResult>.IsHalting(new Func<SelfApplicableFunc<TResult>, TResult>(f))(f))
                (_ => Functions<TResult, Boolean>.Sequence(OmegaCombinators<TResult>.Ω)(False))
                (_ => True);
}

Here since f can be applied with itself, it is represented with the SelfApplicableFunc<TResult> function type.

It is interesting when IsNotHalting is applied with argument IsNotHalting (itself). Assume IsNotHalting halts with IsNotHalting, in another word:

IsHalting IsNotHalting IsNotHalting
≡ True

then there is:

IsNotHalting IsNotHalting
≡ If (IsHalting IsNotHalting IsNotHalting) (λx.Sequence Ω False) (λx.True)
≡ If (True) (λx.Sequence Ω False) (λx.True)
≡ Sequence Ω False
≡ Sequence (ω ω) False
≡ Sequence ((λx.x x) (λx.x x)) False
≡ Sequence ((λx.x x) (λx.x x)) False
≡ ...

So IsNotHalting IsNotHalting is reduced to Sequence Ω False, and is then reduced forever, which means actually IsNotHalting does not halt with IsNotHalting.

On the other hand, Assume IsNotHalting does not halt with IsNotHalting, in another word:

IsHalting IsNotHalting IsNotHalting
≡ False

then there is:

IsNotHalting IsNotHalting
≡ If (IsHalting IsNotHalting IsNotHalting) (λx.Sequence Ω False) (λx.True)
≡ If (False) (λx.Sequence Ω False) (λx.True)
≡ True

So IsNotHalting IsNotHalting is reduced to True, which means IsNotHalting halts with IsNotHalting.

Therefore, if IsHalting exists, it leads to IsNotHalting with the following properties:

If IsNotHalting halts with IsNotHalting, then IsNotHalting does not halt with IsNotHalting
If IsNotHalting does not halt with IsNotHalting, then IsNotHalting halts with IsNotHalting.

This proves IsNotHalting and IsHalting cannot exist.

Equivalence problem

After understanding the halting problem, the equivalence problem becomes very easy to prove. Assume an AreEquivalent function exists:

AreEquivalent := λa.λb.If (/* a and b are equivalent */) (λx.True) (λx.False)

which takes 2 lambda expression as parameter, and returns True/False if they are/are not equivalent. Now define the following 2 functions:

GetTrue1 := λf.λx.λy.Sequence (f x) True
GetTrue2 := λf.λx.λy.True

Given arbitrary function f and its argument x:

GetTrue1 f x
≡ λy.Sequence (f x) True

  GetTrue2 f x
≡ λy.True

For specified f and x:

if f halts with x, then, ∀y, (GetTrue1 f x y) and (GetTrue2 f x y) both always returns True. That is, partially applied functions GetTrue1 f x and GetTrue2 f x are equivalent.
if f does not halt with x, then, ∀y, (GetTrue1 f x y) never returns True, and (GetTrue2 f x y) always returns True. That is, partially applied functions (GetTrue1 f x) and (GetTrue2 f x) are not equivalent.

Now halting problem and equivalence problem are connected. IsHalting function can be directly defined by AreEquivalent function:

IsHalting := λf.λx.AreEquivalent (GetTrue1 f x) (GetTrue2 f x)

The partial application (GetTrue1 f x) and (GetTrue2 f x) can be substituted as:

IsHalting := λf.λx.AreEquivalent (λy.Sequence (f x) True) (λy.True)

In C#:

internal static class Equivalence<T, TResult>
{
    // IsEquivalent = f1 => f2 => True if f1 and f2 are equivalent; otherwise, False
    internal static readonly Func<Func<T, TResult>, Func<Func<T, TResult>, Boolean>>
        IsEquivalent = f1 => f2 => throw new NotImplementedException();

    // IsHalting = f => x => IsEquivalent(_ => Sequence(f(x))(True))(_ => True)
    internal static readonly Func<Func<T, TResult>, Func<T, Boolean>>
        IsHalting = f => x => Equivalence<T, Boolean>.IsEquivalent(_ => Functions<TResult, Boolean>.Sequence(f(x))(True))(_ => True);
}

If the above AreEquivalent function can be defined, then IsHalting can be defined. It is already approved that IsHalting cannot exist, so AreEquivalent cannot exist either. This demonstrates equivalence problem is just another version of halting problem. So, lambda expressions’ equivalence is undecidable. The undecidability is actually a very general topic in computability theory and mathematical logic. The undecidability of halting problem and lambda calculus’ undecidability of equivalence are examples of Rice's theorem, and also examples of Kurt Gödel's incompleteness theorems.

Lambda Calculus via C# (7) Fixed Point Combinator and Recursion

Sat, 23 Nov 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Lambda Calculus via C# series]

p is the fixed point (aka invariant point) of function f if and only if:

p
≡ f p

Take function Math.Sqrt as example, it has 2 fix point, 0 and 1, so that 0 ≡ Math.Sqrt(0) and 1 ≡ Math.Sqrt(1).

The above fixed point definition also leads to infinite substitution:

p
≡ f p
≡ f (f p)
≡ f (f (f p))
≡ ...
≡ f (f (f ... (f p) ...))

Similarly, the fixed point combinator Y is defined as if Y f is the fixed point of f:

(Y f)
≡ f (Y f)

Normal order fixed point combinator (Y combinator) and recursion

The following Y combinator is an implementation of fixed point combinator, discovered by Haskell Curry:

Y := λf.(λg.f (g g)) (λg.f (g g))

It is called the normal order fixed point combinator:

Y f
≡ (λf.(λg.f (g g)) (λg.f (g g))) f
≡ (λg.f (g g)) (λg.f (g g))
≡ f ((λg.f (g g)) (λg.f (g g)))
≡ f (Y f)

The following is Y implemented in SKI:

Y := S (K (S I I)) (S (S (K S) K) (K (S I I)))

And just in SK:

Y := S S K (S (K (S S (S (S S K)))) K)

When Y f can also be substituted infinitely:

(Y f)
≡ f (Y f)
≡ f (f (Y f))
≡ f (f (f (Y f)))
≡ ...
≡ f (f (f ... (f (Y f)) ...))

So Y can be used to implement recursion. As fore mentioned, in lambda calculus, a function cannot directly apply it self in its body. Take the factorial function as example, the factorial of n is defined recursively:

If n is greater than 0, then factorial of n is the multiplication of n and factorial of n – 1
if n is 0, then factorial of n is 1

So naturally:

Factorial := λn.If (n == 0) (λx.1) (λx.n * (Factorial (n - 1)))

However, in lambda calculus the above definition is illegal, because the self reference does not work anonymously:

λn.If (n == 0) (λx.1) (λx.n * (? (n - 1)))

Now with the power of Y combinator, the recursion can be implemented, but still in the anonymous way. First, in above definition, just pass the reference of itself as an variable/argument:

λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))

If the above function is called FactorialHelper, then the Factorial function can be implemented as:

FactorialHelper := λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))
Factorial := Y FactorialHelper

So the recursive Factorial is implemented anonymously:

Factorial
≡ Y FactorialHelper
≡ (λf.(λg.f (g g)) (λg.f (g g))) FactorialHelper
≡ (λf.(λg.f (g g)) (λg.f (g g))) (λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1))))

When Factorial is applied, according to the definition of Factorial and Y:

Factorial 3
≡ Y FactorialHelper 3
≡ FactorialHelper (Y FactorialHelper) 3

Here (Y FactorialHelper) can be substituted by Factorial, according to the definition. So FactorialHelper is called with Factorial and n, exactly as expected.

The normal order Y combinator does not work with applicative order reduction. In applicative order, here FactorialHelper is applied with (Y FactorialHelper), so the right most argument Y FactorialHelper should be reduced first, which leads to infinite reduction:

FactorialHelper (Y FactorialHelper) 3
≡ FactorialHelper (FactorialHelper (Y FactorialHelper)) 3
≡ FactorialHelper (FactorialHelper (FactorialHelper (Y FactorialHelper))) 3
≡ ...

The normal order Y combinator only works with normal order. In normal order, here FactorialHelper is applied with (Y FactorialHelper), so the left most function FactorialHelper should be reduced first:

FactorialHelper (Y FactorialHelper) 3
≡ (λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))) (Y FactorialHelper) 3
≡ (λn.If (n == 0) (λx.1) (λx.n * (Y FactorialHelper (n - 1)))) 3
≡ If (3 == 0) (λx.1) (λx.3 * (Y FactorialHelper (3 - 1)))
≡ If (False) (λx.1) (λx.3 * (Y FactorialHelper (3 - 1))
≡ 3 * (Y FactorialHelper (3 - 1))
≡ 3 * (FactorialHelper (Y FactorialHelper) (3 - 1))
≡ 3 * ((λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))) (Y FactorialHelper) (3 - 1))
≡ 3 * ((λn.If (n == 0) (λx.1) (λx.n * (Y FactorialHelper (n - 1)))) (3 - 1))
≡ 3 * (If ((3 - 1) == 0) (λx.1) (λx.(3 - 1) * (Y FactorialHelper ((3 - 1) - 1))))
≡ 3 * ((3 - 1) * (Y FactorialHelper ((3 - 1) - 1)))
≡ 3 * (2 * (Y FactorialHelper ((3 - 1) - 1)))
≡ 3 * (2 * (FactorialHelper (Y FactorialHelper) ((3 - 1) - 1)))
≡ 3 * (2 * ((λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))) (Y FactorialHelper) ((3 - 1) - 1)))
≡ 3 * (2 * ((λn.If (n == 0) (λx.1) (λx.n * (Y FactorialHelper (n - 1)))) ((3 - 1) - 1)))
≡ 3 * (2 * (If (((3 - 1) - 1) == 0) (λx.1) (λx.((3 - 1) - 1) * (Y FactorialHelper (((3 - 1) - 1) - 1)))))
≡ 3 * (2 * (((3 - 1) - 1) * (Y FactorialHelper (((3 - 1) - 1) - 1))))
≡ 3 * (2 * (1 * (Y FactorialHelper (((3 - 1) - 1) - 1))))
≡ 3 * (2 * (1 * (FactorialHelper (Y FactorialHelper) (((3 - 1) - 1) - 1))))
≡ 3 * (2 * (1 * ((f.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))) (Y FactorialHelper) (((3 - 1) - 1) - 1))))
≡ 3 * (2 * (1 * ((n.If (n == 0) (λx.1) (λx.n * (Y FactorialHelper (n - 1)))) (((3 - 1) - 1) - 1))))
≡ 3 * (2 * (1 * (If ((((3 - 1) - 1) - 1) == 0) (λx.1) (λx.(((3 - 1) - 1) - 1) * (Y FactorialHelper ((((3 - 1) - 1) - 1) - 1))))))
≡ 3 * (2 * (1 * 1))

So the Y f infinite reduction is blocked in in normal order reduction. First, Y f is reduced to f (Y f), then the next reduction is to reduce leftmost expression f, not the the rightmost (Y f). In the above example Y FactorialHelper n:

If n is greater than 0, Y Factorial n is reduced to n * (Y Factorial (n - 1)), where Y Factorial can be further reduced, so the recursion continues.
If n is 0, Y Factorial n is reduced to 1. The reduction ends, so the recursion terminates.

Y combinator is easy to implement in C#. Generally, for a recursive function f of type T -> TResult, its helper function accepts the T -> TResult function and a T value, then return TResult, so its helper function is of type (T -> TResult) –> T -> TResult. Y can be viewed as accepting helper function and returns f. so Y is of type ((T -> TResult) –> T -> TResult) -> (T -> TResult). So:

public static partial class FixedPointCombinators<T, TResult>
{
    // Y = (g => f(g(g)))(g => f(g(g)))
    public static readonly Func<Func<Func<T, TResult>, Func<T, TResult>>, Func<T, TResult>>
        Y = f => new SelfApplicableFunc<Func<T, TResult>>(g => f(g(g)))(g => f(g(g)));
}

Here are the types of the elements in above lambda expression:

g: SelfApplicableFunc<T -> TResult>
g(g): T -> TResult
f: (T -> TResult) –> T -> TResult
f(g(g)): T => TResult
g => f(g(g)): SelfApplicableFunc<T -> TResult> –> T -> TResult, which is SelfApplicableFunc<T -> TResult> by definition
(g => f(g(g)))(g => f(g(g))): T -> TResult

For Factorial, apparently it is of function type Numeral -> Numeral, so FactorialHelper is of function type (Numeral -> Numeral) –> Numeral -> Numeral:

using static FixedPointCombinators<Numeral, Numeral>;

public static partial class ChurchNumeral
{
    // FactorialHelper = factorial => n => If(n == 0)(_ => 1)(_ => n * factorial(n - 1))
    public static readonly Func<Func<Numeral, Numeral>, Func<Numeral, Numeral>>
        FactorialHelper = factorial => n =>
            If(n.IsZero())
                (_ => One)
                (_ => n.Multiply(factorial(n.Subtract(One))));

    public static readonly Func<Numeral, Numeral>
        Factorial = Y(FactorialHelper);
}

Calling above Factorial always throws StackOverflowException, because in C# executes in applicative order. When Factorial is called, it calls normal order Y in applicative order, which causes infinite execution.

Applicative order fixed point combinator (Z combinator) and recursion

The above Y combinator does not work in C#. When reducing Y f in applicative order, the self application in expression f (g g) leads to infinite reduction, which need to be blocked. The solution is to eta convert f (g g) to λx.f (g g) x. So the applicative order fixed point combinator is:

Z := λf.(λg.λx.f (g g) x) (λg.λx.f (g g) x)

It is called Z combinator. Now reduce Z f in applicative order:

Z f
≡ (λf.(λg.λx.f (g g) x) (λg.λx.f (g g) x)) f
≡ (λg.λx.f (g g) x) (λg.λx.f (g g) x)
≡ λx.f ((λg.λx.f (g g) x) (λg.λx.f (g g) x)) x
≡ λx.f (Z f) x

This time Z f is not reduced to f (Z f), but reduced to the eta expanded version λx.f (Z f) x, so any further reduction is blocked. Still take factorial as example:

Factorial 3
≡ Z FactorialHelper 3
≡ (λx.FactorialHelper (Z FactorialHelper) x) 3
≡ FactorialHelper (Z FactorialHelper) 3
≡ FactorialHelper (λx.FactorialHelper (Z FactorialHelper) x) 3
≡ (λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))) (λx.FactorialHelper (Z FactorialHelper) x) 3
≡ (λn.If (n == 0) (λx.1) (λx.n * ((λx.FactorialHelper (Z FactorialHelper) x) (n - 1)))) 3
≡ If (3 == 0) (λx.1) (λx.3 * ((λx.FactorialHelper (Z FactorialHelper) x) (3 - 1)))
≡ If (False) (λx.1) (λx.3 * ((λx.FactorialHelper (Z FactorialHelper) x) (3 - 1)))
≡ 3 * ((λx.FactorialHelper (Z FactorialHelper) x) (3 - 1))
≡ 3 * ((λx.FactorialHelper (Z FactorialHelper) x) 2)
≡ 3 * (FactorialHelper (Z FactorialHelper) 2)
≡ 3 * (FactorialHelper (λx.FactorialHelper (Z FactorialHelper) x) 2)
≡ 3 * ((λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))) (λx.FactorialHelper (Z FactorialHelper) x) 2)
≡ 3 * ((λn.If (n == 0) (λx.1) (λx.n * ((λx.FactorialHelper (Z FactorialHelper) x) (n - 1)))) 2)
≡ 3 * (If (2 == 0) (λx.1) (λx.2 * ((λx.FactorialHelper (Z FactorialHelper) x) (2 - 1))))
≡ 3 * (If (False) (λx.1) (λx.2 * ((λx.FactorialHelper (Z FactorialHelper) x) (2 - 1))))
≡ 3 * (2 * ((λx.FactorialHelper (Z FactorialHelper) x) (2 - 1)))
≡ 3 * (2 * ((λx.FactorialHelper (Z FactorialHelper) x) 1))
≡ 3 * (2 * (FactorialHelper (Z FactorialHelper) 1))
≡ 3 * (2 * (FactorialHelper (λx.FactorialHelper (Z FactorialHelper) x) 1))
≡ 3 * (2 * ((λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))) (λx.FactorialHelper (Z FactorialHelper) x) 1))
≡ 3 * (2 * ((λn.If (n == 0) (λx.1) (λx.n * ((λx.FactorialHelper (Z FactorialHelper) x) (n - 1)))) 1))
≡ 3 * (2 * (If (1 == 0) (λx.1) (λx.1 * ((λx.FactorialHelper (Z FactorialHelper) x) (1 - 1)))))
≡ 3 * (2 * (If (False) (λx.1) (λx.1 * ((λx.FactorialHelper (Z FactorialHelper) x) (1 - 1)))))
≡ 3 * (2 * (1 * ((λx.FactorialHelper (Z FactorialHelper) x) (1 - 1))))
≡ 3 * (2 * (1 * ((λx.FactorialHelper (Z FactorialHelper) x) 0)))
≡ 3 * (2 * (1 * (FactorialHelper (Z FactorialHelper) 0)))
≡ 3 * (2 * (1 * (FactorialHelper (λx.FactorialHelper (Z FactorialHelper) x) 0)))
≡ 3 * (2 * (1 * ((λf.λn.If (n == 0) (λx.1) (λx.n * (f (n - 1)))) (λx.FactorialHelper (Z FactorialHelper) x) 0)))
≡ 3 * (2 * (1 * ((λn.If (n == 0) (λx.1) (λx.n * ((λx.FactorialHelper (Z FactorialHelper) x) (n - 1)))) 0)))
≡ 3 * (2 * (1 * (If (0 == 0) (λx.1) (λx.0 * ((λx.FactorialHelper (Z FactorialHelper) x) (n - 1))))))
≡ 3 * (2 * (1 * (If (True) (λx.1) (λx.0 * ((λx.FactorialHelper (Z FactorialHelper) x) (n - 1))))))
≡ 3 * (2 * (1 * 1))

In C#, Z combinator can be implemented in the same pattern. Just eta expand f(g(g)) to x => f(g(g))(x):

public static partial class FixedPointCombinators<T, TResult>
{
    // Z = (g => x => f(g(g))(x))(g => x => f(g(g))(x))
    public static readonly Func<Func<Func<T, TResult>, Func<T, TResult>>, Func<T, TResult>>
        Z = f => new SelfApplicableFunc<Func<T, TResult>>(g => x => f(g(g))(x))(g => x => f(g(g))(x));
}

The types of the elements in above lambda expression are the same as in Y combinator, and x is of type T.

Now Factorial can be defined with Z and above FactorialHelper:

using static ChurchBoolean;
using static FixedPointCombinators<Numeral, System.Func<Numeral, Numeral>>;

public static partial class ChurchNumeral
{
    // DivideByHelper = divideBy => dividend => divisor => If(dividend >= divisor)(_ => 1 + divideBy(dividend - divisor)(divisor))(_ => 0)
    private static readonly Func<Func<Numeral, Func<Numeral, Numeral>>, Func<Numeral, Func<Numeral, Numeral>>> DivideByHelper = divideBy => dividend => divisor =>
            If(dividend.IsGreaterThanOrEqualTo(divisor))
                (_ => One.Add(divideBy(dividend.Subtract(divisor))(divisor)))
                (_ => Zero);

    public static readonly Func<Numeral, Func<Numeral, Numeral>> 
        DivideBy = Z(DivideByHelper);
}

Another recursion example is Fibonacci number. The nth Fibonacci number is defined recursively:

if n is greater than 1, then the nth Fibonacci number is the sum of the (n -1)th Fibonacci number and the (n -2)th Fibonacci number.
if n is 1 or 0, then the nth Fibonacci number is n

So naturally:

Fibonacci := λn.If (n > 1) (λx.(Fibonacci (n - 1)) + (Fibonacci (n - 2))) (λx.n)

Again, the above recursive definition is illegal in lambda calculus, because the self reference does not work anonymously:

λn.If (n > 1) (λx.(? (n - 1)) + (? (n - 2))) (λx.n)

Following the same helper function pattern as FactorialHelper, a FibonacciHelper can be defined to pass the Fibonacci function as a variable/argument, then Fibonacci can be defined with Z and FibonacciHelper:

FibonacciHelper := λf.λn.If (n > 1) (λx.(f (n - 1)) + (f (n - 2))) (λx.n)
Fibonacci := Z FibonacciHelper

Now Fibonacci is recursive but still can go anonymous, without any self reference:

Fibonacci
≡ Z FibonacciHelper
≡ (λf.(λg.λx.f (g g) x) (λg.λx.f (g g) x)) FibonacciHelper
≡ (λf.(λg.λx.f (g g) x) (λg.λx.f (g g) x)) (λf.λn.If (n > 1) (λx.(f (n - 1)) + (f (n - 2))) (λx.n))

In C#:

// FibonacciHelper  = fibonacci  => n => If(n > 1)(_ => fibonacci(n - 1) + fibonacci(n - 2))(_ => n)
private static readonly Func<Func<Numeral, Numeral>, Func<Numeral, Numeral>>
    FibonacciHelper = fibonacci => n =>
        If(n.IsGreaterThan(One))
            (_ => fibonacci(n.Subtract(One)).Add(fibonacci(n.Subtract(Two))))
            (_ => n);

// Fibonacci = Z(FibonacciHelper)
public static readonly Func<Numeral, Numeral>
    Fibonacci = Z(FibonacciHelper);

Previously, in the Church numeral arithmetic, the following illegal DivideBy with self reference was temporarily used:

DivideBy := λa.λb.If (a >= b) (λx.1 + (DivideBy (a - b) b)) (λx.0)

Finally, with Z, an legal DivideBy in lambda calculus can be defined, following the same helper function pattern:

DivideByHelper := λf.λa.λb.If (a >= b) (λx.1 + (f (a - b) b)) (λx.0)
DivideBy := Z DivideByHelper

The following is the formal version of DivideBy:

DivideBy
≡ Z DivideByHelper
≡ (λf.(λg.λx.f (g g) x) (λg.λx.f (g g) x)) DivideByHelper
≡ (λf.(λg.λx.f (g g) x) (λg.λx.f (g g) x)) (λf.λa.λb.If (a >= b) (λx.1 + (f (a - b) b)) (λx.0))

In C#:

// DivideByHelper = divideBy => dividend => divisor => If(dividend >= divisor)(_ => 1 + divideBy(dividend - divisor)(divisor))(_ => 0)
private static readonly Func<Func<Numeral, Func<Numeral, Numeral>>, Func<Numeral, Func<Numeral, Numeral>>>
    DivideByHelper = divideBy => dividend => divisor =>
        If(dividend.IsGreaterThanOrEqualTo(divisor))
            (_ => One.Add(divideBy(dividend.Subtract(divisor))(divisor)))
            (_ => Zero);

// DivideBy = Z(DivideByHelper)
public static readonly Func<Numeral, Func<Numeral, Numeral>>
    DivideBy = Z(DivideByHelper);

The following are a few examples

public static partial class NumeralExtensions
{
    public static Numeral Factorial(this Numeral n) => ChurchNumeral.Factorial(n);

    public static Numeral Fibonacci(this Numeral n) => ChurchNumeral.Fibonacci(n);

    public static Numeral DivideBy(this Numeral dividend, Numeral divisor) => 
        ChurchNumeral.DivideBy(dividend)(divisor);
}

[TestClass]
public partial class FixedPointCombinatorTests
{
    [TestMethod]
    public void FactorialTest()
    {
        Func<uint, uint> factorial = null; // Must have to be compiled.
        factorial = x => x == 0 ? 1U : x * factorial(x - 1U);

        Assert.AreEqual(factorial(0U), 0U.Church().Factorial().Unchurch());
        Assert.AreEqual(factorial(1U), 1U.Church().Factorial().Unchurch());
        Assert.AreEqual(factorial(2U), 2U.Church().Factorial().Unchurch());
        Assert.AreEqual(factorial(8U), 8U.Church().Factorial().Unchurch());
    }

    [TestMethod]
    public void FibonacciTest()
    {
        Func<uint, uint> fibonacci = null; // Must have. So that fibonacci can recursively refer itself.
        fibonacci = x => x > 1U ? fibonacci(x - 1) + fibonacci(x - 2) : x;

        Assert.AreEqual(fibonacci(0U), 0U.Church().Fibonacci().Unchurch());
        Assert.AreEqual(fibonacci(1U), 1U.Church().Fibonacci().Unchurch());
        Assert.AreEqual(fibonacci(2U), 2U.Church().Fibonacci().Unchurch());
        Assert.AreEqual(fibonacci(8U), 8U.Church().Fibonacci().Unchurch());
    }

    [TestMethod]
    public void DivideByTest()
    {
        Assert.AreEqual(1U / 1U, 1U.Church().DivideBy(1U.Church()).Unchurch());
        Assert.AreEqual(1U / 2U, 1U.Church().DivideBy(2U.Church()).Unchurch());
        Assert.AreEqual(2U / 2U, 2U.Church().DivideBy(2U.Church()).Unchurch());
        Assert.AreEqual(2U / 1U, 2U.Church().DivideBy(1U.Church()).Unchurch());
        Assert.AreEqual(8U / 3U, 8U.Church().DivideBy(3U.Church()).Unchurch());
        Assert.AreEqual(3U / 8U, 3U.Church().DivideBy(8U.Church()).Unchurch());
    }
}

Lambda Calculus via C# (6) Combinatory Logic

Tue, 19 Nov 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Lambda Calculus via C# series]

In lambda calculus, the primitive is function, which can have free variables and bound variables. Combinatory logic was introduced by Moses Schönfinkel and Haskell Curry in 1920s. It is equivalent variant lambda calculus, with combinator as primitive. A combinator can be viewed as an expression with no free variables in its body.

Combinator

The following is the simplest function definition expression, with only bound variable and no free variable:

I := λx.x

In combinatory logic it is called I (Id) combinator. The following functions are combinators too:

S := λx.λy.λz.x z (y z)
K := λx.λy.x

Here S (Slider) combinator slides z to between x and y (In some materials S is called Substitution; In presentation of Dana Scott S is called Slider), and K (Killer) combinator kills y.

In C#, just leave the each combinator’s variables as dynamic:

public static partial class SkiCombinators
{
    public static readonly Func<dynamic, Func<dynamic, Func<dynamic, dynamic>>>
        S = x => y => z => x(z)(y(z));

    public static readonly Func<dynamic, Func<dynamic, dynamic>>
        K = x => y => x;

    public static readonly Func<dynamic, dynamic>
        I = x => x;
}

ω is the self application combinator. It applies variable f to f itself:

ω := λf.f f

Just like above f, ω can also be applied with ω itself, which is the definition of Ω:

Ω := ω ω ≡ (λf.f f) (λf.f f)

Here ω is a function definition expression without free variables, and Ω is a function application expression, which contains no free variables. For Ω, its function application can be beta reduced forever:

(λf.f f) (λf.f f)
≡ (λf.f f) (λf.f f)
≡ (λf.f f) (λf.f f)
≡ ...

So ω ω is an infinite application. Ω is called the looping combinator.

In C#, it is easy to define the type of self applicable function, like above f. Assume the function’s return type is TResult, then this function is of type input –> TResult:

public delegate TResult Func<TResult>(?);

The input type is the function type itself, so it is:

public delegate TResult Func<TResult>(Func<TResult> self)

Above Func<TResult> is the self applicable function type. To be unambiguous with System.Func<TResult>, it can be renamed to SelfApplicableFunc<TResult>:

public delegate TResult SelfApplicableFunc<TResult>(SelfApplicableFunc<TResult> self);

So SelfApplicableFunc<TResult> is equivalent to SelfApplicableFunc<TResult> -> TResult. Since f is of type SelfApplicableFunc<TResult>, f(f) returns TResult. And since ω accept f and returns TResult. ω is of type SelfApplicableFunc<TResult> -> TResult, which is the definition of SelfApplicableFunc<TResult>, so ω is still of type SelfApplicableFunc<TResult>, ω(ω) is still of type TResult:

public static class OmegaCombinators<TResult>
{
    public static readonly SelfApplicableFunc<TResult>
        ω = f => f(f);

    public static readonly TResult
        Ω = ω(ω);
}

SKI combinator calculus

The SKI combinator calculus is a kind of combinatory logic. As a variant of lambda calculus, SKI combinatory logic has no general expression definition rules, or general expression reduction rules. It only has the above S, K, I combinators as the only 3 primitives, and the only 3 function application rules. It can be viewed as a reduced version of lambda calculus, and an extremely simple Turing complete language with only 3 elements: S, K, I.

Take the Boolean values as a simple example. Remember in lambda calculus, True and False are defined as:

True := λt.λf.t
False := λt.λf.f

So that when they are applied:

True t f
≡ (λt.λf.t) t f
≡ t

  False t f
≡ (λt.λf.f) t f
≡ f

Here in SKI combinator calculus, SKI combinators are the only primitives, so True and False can be defined as:

True := K
False := S K

So that when they are applied, they return the same result as the lambda calculus definition:

True t f
≡ K t f
≡ t

  False t f
≡ S K t f
≡ K f (t f) 
≡ f

Remember function composition is defined as:

(f2 ∘ f1) x := f2 (f1 x)

In SKI, the composition operator can be equivalently defined as:

Compose := S (K S) K

And this is how it works:

Compose f2 f1 x
≡ S (K S) K f2 f1 x
≡ (K S) f2 (K f2) f1 x
≡ S (K f2) f1 x
≡ (K f2) x (f1 x)
≡ f2 (f1 x)

In lambda calculus, numerals are defined as:

0 := λf.λx.x
1 := λf.λx.f x
2 := λf.λx.f (f x)
3 := λf.λx.f (f (f x))
...

In SKI, numerals are equivalently defined as:

0 := K I                     ≡ K I
1 := I                       ≡ I
2 := S Compose I             ≡ S (S (K S) K) I
3 := S Compose (S Compose I) ≡ S (S (K S) K) (S (S (K S) K) I)
...

When these numerals are applied, they return the same results as lambda calculus definition:

0 f x
≡ K I f x
≡ I x
≡ x

  1 f x
≡ I f x
≡ f x

  2 f x
≡ S Compose I f x
≡ Compose f (I f) x
≡ Compose f f x
≡ f (f x)

  3 f x
≡ S Compose (S Compose I) f x
≡ Compose f (S Compose I f) x
≡ Compose f (Compose f f) x
≡ f (f (f x))

...

In SKI, the self application combinator ω is:

ω := S I I

When it is applied with f, it returns f f:

S I I f
≡ I x (I f) 
≡ f f

So naturally, Ω is defined as:

Ω := (S I I) (S I I)

And it is infinite as in lambda calculus:

S I I (S I I)
≡ I (S I I) (I (S I I)) 
≡ I (S I I) (S I I) 
≡ S I I (S I I)
...

Actually, I combinator can be defined with S and K in either of the following ways:

I := S K K
I := S K S

And they work the same:

I x
≡ S K K x
≡ K x (K x)
≡ x

  I x
≡ S K S x
≡ K x (S x)
≡ x

So I is just a syntactic sugar in SKI calculus.

In C#, these combinators can be implemented as:

using static SkiCombinators;

public static partial class SkiCalculus
{
    public static readonly Boolean
        True = new Boolean(K);

    public static readonly Boolean
        False = new Boolean(S(K));

    public static readonly Func<dynamic, dynamic>
        Compose = S(K(S))(K);

    public static readonly Func<dynamic, dynamic>
        Zero = K(I);

    public static readonly Func<dynamic, dynamic>
        One = I;

    public static readonly Func<dynamic, dynamic>
        Two = S(Compose)(I);

    public static readonly Func<dynamic, dynamic>
        Three = S(Compose)(S(Compose)(I));

    // ...

    public static readonly Func<dynamic, Func<dynamic, dynamic>>
        Increase = S(Compose);

    public static readonly Func<dynamic, dynamic>
        ω = S(I)(I);

    public static readonly Func<dynamic, dynamic>
        Ω = S(I)(I)(S(I)(I));

    public static readonly Func<dynamic, dynamic>
        IWithSK = S(K)(K); // Or S(K)(S).
}

SKI compiler: compile lambda calculus expression to SKI calculus combinator

The S, K, I combinators can be composed to new combinator that equivalent to any lambda calculus expression. An arbitrary expression in lambda calculus can be converted to combinator in SKI calculus. Assume v is a variable in lambda calculus, and E is an expression in lambda calculus, the conversion ToSki is defined as:

ToSki (v) => v
ToSki (E1 E2) => (ToSki (E1) (ToSki (E2)))
ToSki (λv.E) => (K (ToSki (E))), if x does not occur free in E
ToSki (λv.v) => I
ToSki (λv1.λv2.E) => ToSki (λv1.ToSki (λv2.E))
ToSki (λv.(E1 E2)) => (S (ToSki (λ.v.E1)) (ToSki (λv.E2)))

Based on these rules, a compiler can be implemented to compile a expression in lambda calculus to combinator in SKI calculus. As mentioned before, the C# lambda expression can be compiled as function, and also expression tree data representing the logic of that function:

internal static void FunctionAsData<T>()
{
    Func<T, T> idFunction = value => value;
    Expression<Func<T, T>> idExpression = value => value;
}

The above idFunction and idExpression shares the same lambda expression syntax, but is executable function, while the idExpression is a abstract syntax tree data structure, representing the logic of idFunction:

Expression<Func<T, T>> (NodeType = Lambda, Type = Func<T, T>)
|_Parameters
| |_ParameterExpression (NodeType = Parameter, Type = T)
|   |_Name = "value"
|_Body
  |_ParameterExpression (NodeType = Parameter, Type = T)
    |_Name = "value"

This metaprogramming feature provides great convenience for the conversion – just build the lambda calculus expression as .NET expression tree, traverse the tree and apply the above rules, and convert the tree to another tree representing the SKI calculus combinator.

A SKI calculus combinator, like above Ω combinator (S I I) (S I I), is a composition of S, K, I. The S, K, I primitives can be represented with a constant expression:

public class CombinatorExpression : Expression
{
    private CombinatorExpression(string name) => this.Name = name;

    public static CombinatorExpression S { get; } = new CombinatorExpression(nameof(S));

    public static CombinatorExpression K { get; } = new CombinatorExpression(nameof(K));

    public static CombinatorExpression I { get; } = new CombinatorExpression(nameof(I));

    public string Name { get; }

    public override ExpressionType NodeType { get; } = ExpressionType.Constant;

    public override Type Type { get; } = typeof(object);
}

The composition can be represented with a function application expression:

public class ApplicationExpression : Expression
{
    internal ApplicationExpression(Expression function, Expression variable)
    {
        this.Function = function;
        this.Variable = variable;
    }

    public Expression Function { get; }

    public Expression Variable { get; }

    public override ExpressionType NodeType { get; } = ExpressionType.Invoke;

    public override Type Type { get; } = typeof(object);
}

So the above Ω combinator (S I I) (S I I) can be represented by the following expression tree:

ApplicationExpression (NodeType = Invoke, Type = object)
|_Function
| |_ApplicationExpression (NodeType = Invoke, Type = object)
|   |_Function
|   | |_ApplicationExpression (NodeType = Invoke, Type = object)
|   |   |_Function
|   |   | |_CombinatorExpression (NodeType = Constant, Type = object)
|   |   |   |_Name = "S"
|   |   |_Variable
|   |     |_CombinatorExpression (NodeType = Constant, Type = object)
|   |       |_Name = "I"
|   |_Variable
|     |_CombinatorExpression (NodeType = Constant, Type = object)
|       |_Name = "I"
|_Variable
  |_ApplicationExpression (NodeType = Invoke, Type = object)
    |_Function
    | |_ApplicationExpression (NodeType = Invoke, Type = object)
    |   |_Function
    |   | |_CombinatorExpression (NodeType = Constant, Type = object)
    |   |   |_Name = "S"
    |   |_Variable
    |     |_CombinatorExpression (NodeType = Constant, Type = object)
    |       |_Name = "I"
    |_Variable
      |_CombinatorExpression (NodeType = Constant, Type = object)
        |_Name = "I"

So in the following SkiCompiler type, the ToSki is implemented to traverse the input abstract syntax tree recursively, and apply the above conversion rules:

public static partial class SkiCompiler
{
    public static Expression ToSki(this Expression lambdaCalculus)
    {
        // Ignore type convertion specified in code or generated by C# compiler.
        lambdaCalculus = lambdaCalculus.IgnoreTypeConvertion();

        switch (lambdaCalculus.NodeType)
        {
            case ExpressionType.Constant:
                // 0. ToSki(S) = S, ToSki(K) = K, ToSki(I) = I.
                if (lambdaCalculus is CombinatorExpression)
                {
                    return lambdaCalculus;
                }
                break;

            case ExpressionType.Parameter:
                // 1. ToSki(v) = v.
                return lambdaCalculus;

            case ExpressionType.Invoke:
                // 2. ToSki(E1(E2)) = ToSki(E1)(ToSKi(E2)).
                ApplicationExpression application = lambdaCalculus.ToApplication();
                return new ApplicationExpression(ToSki(application.Function), ToSki(application.Variable));

            case ExpressionType.Lambda:
                LambdaExpression function = (LambdaExpression)lambdaCalculus;
                ParameterExpression variable = function.Parameters.Single();
                Expression body = function.Body.IgnoreTypeConvertion();

                // 3. ToSki(v => E) = K(ToSki(E)), if v does not occur free in E.
                if (!variable.IsFreeIn(body))
                {
                    return new ApplicationExpression(CombinatorExpression.K, ToSki(body));
                }

                switch (body.NodeType)
                {
                    case ExpressionType.Parameter:
                        // 4. ToSki(v => v) = I
                        if (variable == (ParameterExpression)body)
                        {
                            return CombinatorExpression.I;
                        }
                        break;

                    case ExpressionType.Lambda:
                        // 5. ToSki(v1 => v2 => E) = ToSki(v1 => ToSki(v2 => E)), if v1 occurs free in E.
                        LambdaExpression bodyFunction = (LambdaExpression)body;
                        if (variable.IsFreeIn(bodyFunction.Body))
                        {
                            return ToSki(Expression.Lambda(ToSki(bodyFunction), variable));
                        }
                        break;

                    case ExpressionType.Invoke:
                        // 6. ToSki(v => E1(E2)) = S(ToSki(v => E1))(ToSki(v => E2)).
                        ApplicationExpression bodyApplication = body.ToApplication();
                        return new ApplicationExpression(
                            new ApplicationExpression(
                                CombinatorExpression.S,
                                ToSki(Expression.Lambda(bodyApplication.Function, variable))),
                            ToSki(Expression.Lambda(bodyApplication.Variable, variable)));
                }
                break;
        }
        throw new ArgumentOutOfRangeException(nameof(lambdaCalculus));
    }
}

It calls a few helper functions:

private static Expression IgnoreTypeConvertion(this Expression lambdaCalculus) =>
    lambdaCalculus.NodeType == ExpressionType.Convert
        ? ((UnaryExpression)lambdaCalculus).Operand
        : lambdaCalculus;

private static ApplicationExpression ToApplication(this Expression expression)
{
    switch (expression)
    {
        case ApplicationExpression application:
            return application;
        case InvocationExpression invocation:
            return new ApplicationExpression(invocation.Expression, invocation.Arguments.Single());
    }
    throw new ArgumentOutOfRangeException(nameof(expression));
}

private static bool IsFreeIn(this ParameterExpression variable, Expression lambdaCalculus)
{
    // Ignore type convertion specified in code or generated by C# compiler.
    lambdaCalculus = lambdaCalculus.IgnoreTypeConvertion();

    switch (lambdaCalculus.NodeType)
    {
        case ExpressionType.Invoke:
            ApplicationExpression application = lambdaCalculus.ToApplication();
            return variable.IsFreeIn(application.Function) || variable.IsFreeIn(application.Variable);
        case ExpressionType.Lambda:
            LambdaExpression function = (LambdaExpression)lambdaCalculus;
            return variable != function.Parameters.Single() && variable.IsFreeIn(function.Body);
        case ExpressionType.Parameter:
            return variable == (ParameterExpression)lambdaCalculus;
        case ExpressionType.Constant:
            return false;
    }
    throw new ArgumentOutOfRangeException(nameof(lambdaCalculus));
}

Sometimes, in order to make the lambda calculus expression be compiled, some type information has to be added manually or automatically by C# compiler. These type conversion information is not needed, and can be removed by IgnoreTypeConvertion. In lambda expression, function invocation is compiled as InvocationExpression node with node type Invoke, which is the same as ApplicationExpression. For convenience, ToApplication unifies all Invoke nodes to ApplicationExpression. And IsFreeIn recursively test whether the specified variable occurs free in the specified lambda calculus expression.

Finally, for readability, the following ToSkiString method Converts the compiled SKI calculus expression to string representation:

public static string ToSkiString(this Expression skiCalculus) => skiCalculus.ToSkiString(false);

private static string ToSkiString(this Expression skiCalculus, bool parentheses)
{
    switch (skiCalculus.NodeType)
    {
        case ExpressionType.Invoke:
            ApplicationExpression application = (ApplicationExpression)skiCalculus;
            return parentheses
                ? $"({application.Function.ToSkiString(false)} {application.Variable.ToSkiString(true)})"
                : $"{application.Function.ToSkiString(false)} {application.Variable.ToSkiString(true)}";
        case ExpressionType.Parameter:
            return ((ParameterExpression)skiCalculus).Name;
        case ExpressionType.Constant:
            return ((CombinatorExpression)skiCalculus).Name;
    }
    throw new ArgumentOutOfRangeException(nameof(skiCalculus));
}

The following example demonstrates how to represent 2-tuple in SKI calculus combinator:

internal static void Tuple<T1, T2>()
{
    Expression<Func<T1, Func<T2, Tuple<T1, T2>>>>
        createTupleLambda = item1 => item2 => f => f(item1)(item2);
    Expression createTupleSki = createTupleLambda.ToSki();
    createTupleSki.ToSkiString().WriteLine();
    // S (S (K S) (S (K K) (S (K S) (S (K (S I)) (S (K K) I))))) (K (S (K K) I))
}

To verify the result, a tuple can be created with x as first item, and y as the second item:

CreateTuple x y
≡ S (S (K S) (S (K K) (S (K S) (S (K (S I)) (S (K K) I))))) (K (S (K K) I)) x y
≡ S (K S) (S (K K) (S (K S) (S (K (S I)) (S (K K) I)))) x (K (S (K K) I) x) y
≡ K S x (S (K K) (S (K S) (S (K (S I)) (S (K K) I))) x) (K (S (K K) I) x) y
≡ S (S (K K) (S (K S) (S (K (S I)) (S (K K) I))) x) (K (S (K K) I) x) y
≡ S (K K) (S (K S) (S (K (S I)) (S (K K) I))) x y (K (S (K K) I) x y)
≡ K K x (S (K S) (S (K (S I)) (S (K K) I)) x) y (K (S (K K) I) x y)
≡ K (S (K S) (S (K (S I)) (S (K K) I)) x) y (K (S (K K) I) x y)
≡ S (K S) (S (K (S I)) (S (K K) I)) x (K (S (K K) I) x y)
≡ K S x (S (K (S I)) (S (K K) I) x) (K (S (K K) I) x y)
≡ S (S (K (S I)) (S (K K) I) x) (K (S (K K) I) x y)
≡ S (K (S I) x (S (K K) I x)) (K (S (K K) I) x y)
≡ S (S I (S (K K) I x)) (K (S (K K) I) x y)
≡ S (S I ((K K) x (I x))) (K (S (K K) I) x y)
≡ S (S I (K (I x))) (K (S (K K) I) x y)
≡ S (S I (K x)) (K (S (K K) I) x y)
≡ S (S I (K x)) (S (K K) I y)
≡ S (S I (K x)) (K K y (I y))
≡ S (S I (K x)) (K (I y))
≡ S (S I (K x)) (K y)

To get the first/second item of the above tuple, apply it with True/False:

Item1 (CreateTuple x y)
≡ (CreateTuple x y) True
≡ S (S I (K x)) (K y) True
≡ S (S I (K x)) (K y) K
≡ S I (K x) K (K y K)
≡ I K (K x K) (K y K)
≡ K (K x K) (K y K)
≡ K x K
≡ x

  Item2 (CreateTuple x y)
≡ (CreateTuple x y) False
≡ S (S I (K x)) (K y) False
≡ S (S I (K x)) (K y) (S K)
≡ S I (K x) (S K) (K y (S K))
≡ I (S K) (K x (S K)) (K y (S K))
≡ S K (K x (S K)) (K y (S K))
≡ K y (K x (S K) y)
≡ y

So the compiled 2-tuple SKI calculus combinator is equivalent to the lambda calculus expression.

Another example is the logic operator And:

And := λa.λb.a b False ≡ λa.λb.a b (λt.λf.f)

So in C#:

internal static void And()
{
    Expression<Func<Boolean, Func<Boolean, Boolean>>>
        andLambda = a => b => a(b)((Boolean)(@true => @false => @false));
    Expression andSki = andLambda.ToSki();
    andSki.ToSkiString().WriteLine();;
}

Unfortunately, above expression tree cannot be compiled, with error CS1963: An expression tree may not contain a dynamic operation. The reason is, Boolean is the alias of Func<dynamic, Func<dynamic, dynamic>>, and C# compiler does not support dynamic operations in expression tree, like calling a(b) here. At compile time, dynamic is just object, so the solution is to replace dynamic with object, and replace Boolean with object –> object -> object, then the following code can be compiled:

internal static void And()
{
    Expression<Func<Func<object, Func<object, object>>, Func<Func<object, Func<object, object>>, Func<object, Func<object, object>>>>>
        andLambda = a => b => (Func<object, Func<object, object>>)a(b)((Func<object, Func<object, object>>)(@true => @false => @false));
    Expression andSki = andLambda.ToSki();
    andSki.ToSkiString().WriteLine();
    // S (S (K S) (S (S (K S) (S (K K) I)) (K I))) (K (K (K I)))
}

The compilation result can be verified in similar way:

And True True
≡ S (S (K S) (S (S (K S) (S (K K) I)) (K I))) (K (K (K I))) True True
≡ S (S (K S) (S (S (K S) (S (K K) I)) (K I))) (K (K (K I))) K K
≡ S (K S) (S (S (K S) (S (K K) I)) (K I)) K (K (K (K I)) K) K
≡ K S K (S (S (K S) (S (K K) I)) (K I) K) (K (K (K I)) K) K
≡ S (S (S (K S) (S (K K) I)) (K I) K) (K (K (K I)) K) K
≡ S (S (K S) (S (K K) I)) (K I) K K (K (K (K I)) K K)
≡ S (K S) (S (K K) I) K (K I K) K (K (K (K I)) K K)
≡ K S K (S (K K) I K) (K I K) K (K (K (K I)) K K)
≡ S (S (K K) I K) (K I K) K (K (K (K I)) K K)
≡ S (K K) I K K (K I K K) (K (K (K I)) K K)
≡ K K K (I K) K (K I K K) (K (K (K I)) K K)
≡ K (I K) K (K I K K) (K (K (K I)) K K)
≡ I K (K I K K) (K (K (K I)) K K)
≡ K (K I K K) (K (K (K I)) K K)
≡ K I K K
≡ I K
≡ K
≡ True

  And True False
≡ S (S (K S) (S (S (K S) (S (K K) I)) (K I))) (K (K (K I))) True False
≡ S (S (K S) (S (S (K S) (S (K K) I)) (K I))) (K (K (K I))) K (S K)
≡ (S (K S)) (S (S (K S) (S (K K) I)) (K I)) K (K (K (K I)) K) (S K)
≡ K S K (S (S (K S) (S (K K) I)) (K I) K) (K (K (K I)) K) (S K)
≡ S (S (S (K S) (S (K K) I)) (K I) K) (K (K (K I)) K) (S K)
≡ S (S (K S) (S (K K) I)) (K I) K (S K) (K (K (K I)) K (S K))
≡ S (K S) (S (K K) I) K (K I K) (S K) (K (K (K I)) K (S K))
≡ K S K (S (K K) I K) (K I K) (S K) (K (K (K I)) K (S K))
≡ S (S (K K) I K) (K I K) (S K) (K (K (K I)) K (S K))
≡ S (K K) I K (S K) (K I K (S K)) (K (K (K I)) K (S K))
≡ K K K (I K) (S K) (K I K (S K)) (K (K (K I)) K (S K))
≡ K (I K) (S K) (K I K (S K)) (K (K (K I)) K (S K))
≡ I K (K I K (S K)) (K (K (K I)) K (S K))
≡ K (K I K (S K)) (K (K (K I)) K (S K))
≡ K I K (S K)
≡ I (S K)
≡ S K
≡ False

...

Iota combinator calculus

Another interesting example of combinator logic is Iota combinator calculus. It has only one combinator:

ι := λf.f S K ≡ λf.f (λx.λy.λz.x z (y z)) (λx.λy.x)

That’s the whole combinatory logic. It is an esoteric programming language with minimum element – only 1 single element, but still Turing-complete. With Iota combinator, SKI can be implemented as:

S := ι (ι (ι (ι ι)))
K := ι (ι (ι ι))
I := ι ι

So Iota is as Turing-complete as SKI. For example:

I x
≡ ι ι x
≡ (λf.f S K) (λf.f S K) x
≡ (λf.f S K) S K x
≡ (S S K) K x
≡ S K (K K) x
≡ K x ((K K) x)
≡ x

In C#, these combinators can be implemented as:

public static partial class IotaCombinator
{
    public static readonly Func<dynamic, dynamic>
        ι = f => f
            (new Func<dynamic, Func<dynamic, Func<dynamic, dynamic>>>(x => y => z => x(z)(y(z)))) // S
            (new Func<dynamic, Func<dynamic, dynamic>>(x => y => x)); // K
}

public static class IotaCalculus
{
    public static readonly Func<dynamic, Func<dynamic, Func<dynamic, dynamic>>>
        S = ι(ι(ι(ι(ι))));

    public static readonly Func<dynamic, Func<dynamic, dynamic>>
        K = ι(ι(ι(ι)));

    public static readonly Func<dynamic, dynamic>
        I = ι(ι);
}

Lambda Calculus via C# (5) List

Wed, 13 Nov 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Lambda Calculus via C# series]

In lambda calculus and Church encoding, there are various ways to represent a list with anonymous functions.

Tuple as list node

With Church pair, it is easy to model Church list as a linked list, where each list node is a a Church pair (2-tuple) of current node’s value and the next node, So that

CreateListNode := CreateTuple = λv.λn.λf.f v n
ListNode := Tuple = λf.f v n

Here variable v is the value of the current node, so it is the first item of the tuple; And variable n is the next node of the current node, so it is the second item of the tuple:

Value := Item1 = λl.l (λv.λn.v)
Next := Item2 = λl.l (λv.λn.n)

Here variable l is the list node. The C# implementation is similar to tuple and signed numeral, except ListNode<T> function type now has 1 type parameter, which is the type of its value:

// ListNode<T> is the alias of Tuple<T, ListNode<T>>.
public delegate dynamic ListNode<out T>(Boolean f);

public static partial class ChurchList<T>
{
    // Create = value => next => (value, next)
    public static readonly Func<T, Func<ListNode<T>, ListNode<T>>>
        Create = value => next => new ListNode<T>(ChurchTuple<T, ListNode<T>>.Create(value)(next));

    // Value = node => node.Item1()
    public static readonly Func<ListNode<T>, T> 
        Value = node => new Tuple<T, ListNode<T>>(node).Item1();

    // Next = node => node.Item2()
    public static readonly Func<ListNode<T>, ListNode<T>> 
        Next = node => new Tuple<T, ListNode<T>>(node).Item2();
}

Usually, when a list ends, its last node’s next node is flagged as a special null node. Here in lambda calculus, since a node is an anonymous function, the null node is also an anonymous function:

Null := λf.λx.x

And IsNull predicate returns a Church Boolean to indicate whether a list node is null:

IsNull := λl.l (λv.λn.λx.False) True

When IsNull is applied with a null node:

IsNull Null
≡ (λl.l (λv.λn.λx.False) True) (λf.λx.x)
≡ (λf.λx.x) (λv.λn.λx.False) True
≡ (λx.x) True
≡ True

And when IsNull is applied with a non-null node:

IsNull (CreateListNode 0 Null)
≡ IsNull (λf.f 0 Null)
≡ (λl.l (λv.λn.λx.False) True) (λf.f 0 Null)
≡ (λf.f 0 Null) (λv.λn.λx.False) True
≡ (λv.λn.λx.False) 0 Null True
≡ (λn.λx.False) Null True
≡ (λx.False) True
≡ False

The C# implementation is noisy because a lot of type information has to be provided. This is Null:

using static ChurchBoolean;

public static partial class ChurchList<T>
{
    // Null = False;
    public static readonly ListNode<T>
        Null = new ListNode<T>(False);

    // IsNull = node => node(value => next => _ => False)(True)
    public static readonly Func<ListNode<T>, Boolean> 
        IsNull = node => node(value => next => new Func<Boolean, Boolean>(_ => False))(True);
}

And the indexer for list can be easily defined with as a function accepts a start node and a Church numeral i as the specified index. To return the node at the specified index, just call Next function for i times from the start node:

ListNodeAt := λl.λi.i Next l

C#:

public static readonly Func<ListNode<T>, Func<Numeral, ListNode<T>>>
    ListNodeAt = start => index => index(node => Next(node))(start);

The following are the extension methods wrapping the list operators:

public static class ListNodeExtensions
{
    public static T Value<T>(this ListNode<T> node) => ChurchList<T>.Value(node);

    public static ListNode<T> Next<T>(this ListNode<T> node) => ChurchList<T>.Next(node);

    public static Boolean IsNull<T>(this ListNode<T> node) => ChurchList<T>.IsNull(node);

    public static ListNode<T> ListNodeAt<T>(this ListNode<T> start, Numeral index) => ChurchList<T>.ListNodeAt(start)(index);
}

And the following code demonstrates how the list works:

[TestClass]
public class ChurchListTests
{
    [TestMethod]
    public void CreateValueNextTest()
    {
        ListNode<int> node1 = ChurchList<int>.Create(1)(ChurchList<int>.Null);
        ListNode<int> node2 = ChurchList<int>.Create(2)(node1);
        ListNode<int> node3 = ChurchList<int>.Create(3)(node2);
        Assert.AreEqual(1, node1.Value());
        Assert.AreEqual(ChurchList<int>.Null, node1.Next());
        Assert.AreEqual(2, node2.Value());
        Assert.AreEqual(node1, node2.Next());
        Assert.AreEqual(3, node3.Value());
        Assert.AreEqual(node2, node3.Next());
        Assert.AreEqual(node2.Value(), node3.Next().Value());
        Assert.AreEqual(node1.Value(), node3.Next().Next().Value());
        Assert.AreEqual(ChurchList<int>.Null, node3.Next().Next().Next());
        try
        {
            ChurchList<object>.Null.Next();
            Assert.Fail();
        }
        catch (InvalidCastException exception)
        {
            exception.WriteLine();
        }
    }

    [TestMethod]
    public void IsNullTest()
    {
        ListNode<int> node1 = ChurchList<int>.Create(1)(ChurchList<int>.Null);
        ListNode<int> node2 = ChurchList<int>.Create(2)(node1);
        ListNode<int> node3 = ChurchList<int>.Create(3)(node2);
        Assert.IsTrue(ChurchList<object>.Null.IsNull().Unchurch());
        Assert.IsFalse(node1.IsNull().Unchurch());
        Assert.IsFalse(node2.IsNull().Unchurch());
        Assert.IsFalse(node3.IsNull().Unchurch());
        Assert.IsTrue(node1.Next().IsNull().Unchurch());
        Assert.IsFalse(node2.Next().IsNull().Unchurch());
        Assert.IsFalse(node3.Next().IsNull().Unchurch());
    }

    [TestMethod]
    public void IndexTest()
    {
        ListNode<int> node1 = ChurchList<int>.Create(1)(ChurchList<int>.Null);
        ListNode<int> node2 = ChurchList<int>.Create(2)(node1);
        ListNode<int> node3 = ChurchList<int>.Create(3)(node2);
        Assert.AreEqual(node3, node3.NodeAt(0U.Church()));
        Assert.AreEqual(node2, node3.NodeAt(1U.Church()));
        Assert.AreEqual(node1, node3.NodeAt(2U.Church()));
        Assert.IsTrue(node3.NodeAt(3U.Church()).IsNull().Unchurch());
        try
        {
            node3.NodeAt(4U.Church());
            Assert.Fail();
        }
        catch (InvalidCastException exception)
        {
            exception.WriteLine();
        }
    }
}

Aggregate function as list node

Remember the LINQ Aggregate query method accepting a seed and a accumulator function:

TAccumulate Aggregate<TSource, TAccumulate>(this IEnumerable<TSource> source, TAccumulate seed, Func<TAccumulate, TSource, TAccumulate> func);

Assume seed is x, and accumulator function is f:

When source is empty, the aggregation result is x
When source is { 0 }, the aggregation result is f(x, 0)
When source is { 1, 0 }, the aggregation result is f(f(x, 1), 0)
When source is { 2, 1, 0 }, the aggregation result is f(f(f(x, 2), 1), 0)

Church list can also be encoded with a similar Aggregate function with seed and accumulator function:

dynamic AggregateListNode<T>(dynamic x, Func<dynamic, T, dynamic> f);

Its type parameter T is the type of node value. And since the seed can be anything, just leave it as dynamic as usual. So the list node is of above aggregate function type (dynamic, (dynamic , T) -> dynamic) -> dynamic. After currying the aggregate function and the accumulator function, it becomes dynamic -> (dynamic –> T -> dynamic) -> dynamic. So this is the function type of list node, and an alias can be defined as:

// Curried from: (dynamic, dynamic -> T -> dynamic) -> dynamic.
// AggregateListNode is the alias of: dynamic -> (dynamic -> T -> dynamic) -> dynamic.
public delegate Func<Func<dynamic, Func<T, dynamic>>, dynamic> AggregateListNode<out T>(dynamic x);

And this is the creation and definition of list node:

CreateListNode := λv.λn.λx.λf.f (n x f) v
ListNode := λx.λf.f (n x f) v

In C#:

public static partial class ChurchAggregateList<T>
{
    public static readonly Func<T, Func<AggregateListNode<T>, AggregateListNode<T>>>
        Create = value => next => x => f => f(next(x)(f))(value);
}

Similarly, here variable v is the value of current node, variable n is the next node of the current node. And variable x is the seed for aggregation, variable f is the accumulator function. The list is still modeled as a linked list, so Null is also needed to represent the end of list:

Null := λx.λf.x

Null is defined to call f for 0 times. For example, to create a linked list { 2, 1, 0 }, first create the last list node, with value 2 and Null as its next node:

CreateListNode 0 Null
≡ (λv.λn.λx.λf.f (n x f) v) 0 (λx.λf.x)
≡ (λn.λx.λf.f (n x f) 0) (λx.λf.x)
≡ λx.λf.f ((λx.λf.x) x f) 0
≡ λx.λf.f x 0

Then the previous node can be created with value 1 and the above node:

CreateListNode 1 (CreateListNode 0 Null)
≡ CreateListNode 1 (λx.λf.f x 0)
≡ (λv.λn.λx.λf.f (n x f) v) 1 (λx.λf.f x 0)
≡ (λn.λx.λf.f (n x f) 1) (λx.λf.f x 0)
≡ λx.λf.f ((λx.λf.f x 0) x f) 1
≡ λx.λf.f (f x 0) 1

And the first node has value 0:

CreateListNode 2 (CreateListNode 1 (CreateListNode 0 Null))
≡ CreateListNode 2 (λx.λf.f (f x 0) 1)
≡ (λv.λn.λx.λf.f (n x f) v) 2 (λx.λf.f (f x 0) 1)
≡ (λn.λx.λf.f (n x f) 2) (λx.λf.f (f x 0) 1)
≡ λx.λf.f (λx.λf.f (f x 0) 1) x f) 2
≡ λx.λf.f (f (f x 0) 1) 2

So the list nodes are represented in the same pattern as LINQ aggregation.

The IsNull predicate can be defined as following:

IsNull := λl.l True (λx.λv.False)

The variable l is the list node, which is an aggregate function, and is applied with seed True and accumulate function λv.λx.False. When IsNull is applied with a null node, the accumulate function is not applied, and seed True is directly returned:

IsNull Null
≡ (λl.l True (λx.λv.False)) (λx.λf.x)
≡ (λx.λf.x) True (λx.λv.False)
≡ (λf.True) (λx.λv.False)
≡ True

And when IsNull is applied with a non null node, the accumulator function is applied and constantly returns False, so IsNull returns False:

IsNull (CreateListNode 2 Null)
≡ IsNull (λx.λf.f x 2)
≡ (λl.l True (λx.λv.False)) (λx.λf.f x 2)
≡ (λx.λf.f x 2) True (λx.λv.False)
≡ (λf.f True 2) (λx.λv.False)
≡ (λx.λv.False) True 2
≡ False

In C#:

using static ChurchBoolean;

public static partial class ChurchAggregateList<T>
{
    public static readonly AggregateListNode<T>
        Null = x => f => x;

    public static readonly Func<AggregateListNode<T>, Boolean>
        IsNull = node => node(True)(x => value => False);
}

The following function returns the value from the specified node:

Value := λl.l Id (λx.λv.v)

When Value is applied with a node, which has value v and next node n:

Value (CreateListNode v n)
≡ Value (λx.λf.f (n x f) v)
≡ (λl.l Id (λx.λv.v)) (λx.λf.f (n x f) v)
≡ (λx.λf.f (n x f) v) Id (λx.λv.v)
≡ (λf.f (n Id f) v) (λx.λv.v)
≡ (λx.λv.v) (n Id f) v
≡ (λv.v) v
≡ v

In C#:

// Value = node => node(Id)(x => value => value)
public static readonly Func<AggregateListNode<T>, T>
    Value = node => node(Functions<T>.Id)(x => value => value);

It is not very intuitive to get a node’s next node:

Next := λl.λx.λf.l (λf.x) (λx.λv.λg.g (x f) v) (λx.λv.v)

In C#:

// Next = node => x => f => node(_ => x)(accumulate => value => (g => g(accumulate(f))(value)))(accumulate => value => accumulate);
public static readonly Func<AggregateListNode<T>, AggregateListNode<T>>
    Next = node => x => f => node(new Func<Func<dynamic, Func<T, dynamic>>, dynamic>(_ => x))(accumulate => value => new Func<Func<dynamic, Func<T, dynamic>>, dynamic>(g => g(accumulate(f))(value)))(new Func<dynamic, Func<T, dynamic>>(accumulate => value => accumulate));

The above definition is similar to the the pattern of initial version Subtract function for Church numeral. So it can be defined by shifting tuple too. Again, list node with value v and next node n is a aggregate function, it can be applied with a tuple of Null nodes as seed, and a accumulator function to swap the tuple:

(CreateListNode v n) (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ (λx.λf.f (n x f) v) (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ (λf.f (n (Null, Null) f) v) (λt.λv.Shift (CreateListNode v) t)
≡ (λt.λv.Shift (CreateListNode v) t) (n (Null, Null) (λt.λv.Shift (CreateListNode v)) t) v
≡ (λv.Shift (CreateListNode v) (n (Null, Null) (λt.λv.Shift (CreateListNode v)) t)) v
≡ Shift (CreateListNode v) (n (Null, Null) (λt.λv.Shift (CreateListNode v)) t)

Take list { n, n – 1, …, 2, 1, 0 } as example, assume its nodes are ListNoden, ListNoden - 1, …, ListNode2, ListNode1, ListNode0:

the last node is: CreateListNode 0 Null
the second last node is: CreateListNode 1 (CreateListNode 0 Null)
the third last node is: CreateListNode 2 (CreateListNode 1 (CreateListNode 0 Null))
…

Now apply these nodes with above tuple seed and tuple shifting accumulator function:

ListNode0 (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ (CreateListNode 0 Null) (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ Shift (CreateListNode 0) (Null (Null, Null) (λt.λv.Shift (CreateListNode v)) t)
≡ Shift (CreateListNode 0) ((λx.λf.λx) (Null, Null) (λt.λv.Shift (CreateListNode v)) t)
≡ Shift (CreateListNode 0) (Null, Null)
≡ (Null, CreateListNode 0 Null)
≡ (Null, ListNode0)

  ListNode1 (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ (CreateListNode 1 (CreateListNode 0 Null)) (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ Shift (CreateListNode 1) ((CreateListNode 0 Null) (Null, Null) (λt.λv.Shift (CreateListNode v)) t)
≡ Shift (CreateListNode 1) (Null, Create ListNode 0 Null)
≡ (CreateListNode 0 Null, (CreateListNode 1 (CreateListNode 0 Null))
≡ (ListNode0, ListNode1)

  ListNode2 (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ (CreateListNode 2 (CreateListNode 1 (CreateListNode 0 Null))) (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ Shift (CreateListNode 2) ((CreateListNode 1 (CreateListNode 0 Null)) (Null, Null) (λt.λv.Shift (CreateListNode v)) t)
≡ Shift (CreateListNode 2) (CreateListNode 0 Null, (CreateListNode 1 (CreateListNode 0 Null))
≡ ((CreateListNode 1 (CreateListNode 0 Null), CreateListNode 2 (CreateListNode 1 (CreateListNode 0 Null)))
≡ (ListNode1, ListNode2)

...

  ListNoden (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ (ListNoden - 1, ListNoden)

Generally, there is:

(CreateListNode v n) (Null, Null) (λt.λv.Shift (CreateListNode v) t)
≡ (n, Create v n)

So Next can be defined as:

Next := λl.Item2 (l (CreateTuple Null Null) (λt.λv.Shift (CreateListNode v) t))

In C#:

// Next = node => node((Null, Null))(tuple => value => tuple.Shift(ChurchTuple.Create(value))).Item1()
public static readonly Func<AggregateListNode<T>, AggregateListNode<T>>
    Next = node =>
        ((Tuple<AggregateListNode<T>, AggregateListNode<T>>)node
            (ChurchTuple<AggregateListNode<T>, AggregateListNode<T>>.Create(Null)(Null))
            (tuple => value => ((Tuple<AggregateListNode<T>, AggregateListNode<T>>)tuple).Shift(Create(value))))
        .Item1();

The indexer can be defined the same as above:

ListNodeAt := λl.λi.i Next l

In C#;

public static readonly Func<AggregateListNode<T>, Func<Numeral, AggregateListNode<T>>>
    ListNodeAt = start => index => index(node => Next(node))(start);

The following are the extension methods wrapping the list operators:

public static class AggregateListNodeExtensions
{
    public static Boolean IsNull<T>(this AggregateListNode<T> node) => ChurchAggregateList<T>.IsNull(node);

    public static T Value<T>(this AggregateListNode<T> node) => ChurchAggregateList<T>.Value(node);

    public static AggregateListNode<T> Next<T>(this AggregateListNode<T> node) => 
        ChurchAggregateList<T>.Next(node);

    public static AggregateListNode<T> ListNodeAt<T>(this AggregateListNode<T> start, Numeral index) => 
        ChurchAggregateList<T>.ListNodeAt(start)(index);
}

And the following code demonstrate how the list works:

[TestClass]
public class ChurchAggregateListTests
{
    [TestMethod]
    public void CreateValueNextTest()
    {
        AggregateListNode<int> node1 = ChurchAggregateList<int>.Create(1)(ChurchAggregateList<int>.Null);
        AggregateListNode<int> node2 = ChurchAggregateList<int>.Create(2)(node1);
        AggregateListNode<int> node3 = ChurchAggregateList<int>.Create(3)(node2);
        Assert.AreEqual(1, node1.Value());
        Assert.IsTrue(node1.Next().IsNull().Unchurch());
        Assert.AreEqual(2, node2.Value());
        Assert.AreEqual(node1.Value(), node2.Next().Value());
        Assert.AreEqual(3, node3.Value());
        Assert.AreEqual(node2.Value(), node3.Next().Value());
        Assert.AreEqual(node1.Value(), node3.Next().Next().Value());
        Assert.IsTrue(node3.Next().Next().Next().IsNull().Unchurch());
    }

    [TestMethod]
    public void IsNullTest()
    {
        AggregateListNode<int> node1 = ChurchAggregateList<int>.Create(1)(ChurchAggregateList<int>.Null);
        AggregateListNode<int> node2 = ChurchAggregateList<int>.Create(2)(node1);
        AggregateListNode<int> node3 = ChurchAggregateList<int>.Create(3)(node2);
        Assert.IsTrue(ChurchAggregateList<int>.Null.IsNull().Unchurch());
        Assert.IsFalse(node1.IsNull().Unchurch());
        Assert.IsFalse(node2.IsNull().Unchurch());
        Assert.IsFalse(node3.IsNull().Unchurch());
        Assert.IsTrue(node1.Next().IsNull().Unchurch());
        Assert.IsFalse(node2.Next().IsNull().Unchurch());
        Assert.IsFalse(node3.Next().IsNull().Unchurch());
    }

    [TestMethod]
    public void IndexTest()
    {
        AggregateListNode<int> node1 = ChurchAggregateList<int>.Create(1)(ChurchAggregateList<int>.Null);
        AggregateListNode<int> node2 = ChurchAggregateList<int>.Create(2)(node1);
        AggregateListNode<int> node3 = ChurchAggregateList<int>.Create(3)(node2);
        Assert.AreEqual(node3.Value(), node3.NodeAt(0U.Church()).Value());
        Assert.AreEqual(node2.Value(), node3.NodeAt(1U.Church()).Value());
        Assert.AreEqual(node1.Value(), node3.NodeAt(2U.Church()).Value());
        Assert.IsTrue(node3.NodeAt(3U.Church()).IsNull().Unchurch());
    }
}

Model everything

Once again, in lambda calculus the only primitive is anonymous function. So far many data types and operations are modeled by anonymous functions, including Boolean, unsigned and signed numeral, tuple, list, logic, arithmetic (except division, which will be implemented later), predicate, etc. With these facilities, many other data types and operations can be modeled too. For example:

Floating point number can be represented in the form of significand * baseexponent. In IEEE 754 (aka IEC 60559), floating point numbers are represented as binary format (sign) significand * 2exponent (System.Single and System.Double in .NET), and decimal format (sign) significand * 10exponent (System.Decimal). So either representation can be modeled with a 3-tuple of (Boolean, unsigned numeral, signed numeral).
Character (System.Char in .NET) can be represented by unsigned numeral.
String (System.String in .NET) can be modeled by a list of characters.
Tuple and list can represent other data structures, like tree, stack, queue, etc.
…

And eventually everything can be modeled with anonymous function represented by lambda expression. Actually, lambda calculus is a classic example of Turing completeness. Lambda calculus is introduced by Alonzo Church before Turing machine was introduced by Alan Turing, and they are equivalent. Lambda calculus, as a universal model of computation, is the rationale and foundations of functional programming. Functional languages (or the functional subset of languages) can be viewed as lambda calculus with more specific syntax, and the execution of functional program can be viewed as reduction of lambda calculus expression.

Lambda Calculus via C# (4) Tuple and Signed Numeral

Sun, 10 Nov 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Lambda Calculus via C# series]

Besides modeling values like Boolean and numeral, anonymous function can also model data structures. In Church encoding, Church pair is an approach to use functions to represent a tuple of 2 items.

Church pair (2-tuple)

A tuple can be constructed with its first item x, its second item y, and a function f:

CreateTuple := λx.λy.λf.f x y

So a tuple is can be created by partially applying CreateTuple with 2 items x and y:

Tuple := CreateTuple x y
       ≡ (λx.λy.λf.f x y) x y
       ≡ λf.f x y

So a tuple is a higher-order function, which accept a function f, and apply it with its 2 items. So f accepts 2 arguments, it is in the form of λx.λy.E.

To get the tuple’s first item x, just apply the tuple function with a specific function f, where f simply accepts 2 items and returns the first item:

Tuple (λx.λy.x)
≡ (λf.f x y) (λx.λy.x)
≡ (λx.λy.x) x y
≡ x

Similarly, to get the tuple’s second item y, just apply tuple function with a specific function f, where f simply accepts 2 items and returns the first item:

Tuple (λx.λy.y)
≡ (λf.f x y) (λx.λy.y)
≡ (λx.λy.y) x y
≡ y

So the following Item1 function is defined to accept a tuple, apply the tuple function with function λx.λy.x, and return the tuple’s first item:

Item1 := λt.t (λx.λy.x)

Again, this is how it works:

Item1 (CreateTuple x y)
≡ (λt.t (λx.λy.x)) (CreateTuple x y)
≡ (λt.t (λx.λy.x)) (λf.f x y)
≡ (λf.f x y) (λx.λy.x)
≡ (λx.λy.x) x y
≡ x

And Item2 function can ne defined in the same way to get the tuple’s second item:

Item2 := λt.t (λx.λy.y)

Notice functions λx.λy.x and λx.λy.y can be alpha converted to λt.λf.t and λt.λf.f, which are just Church Boolean True and False. So Item1 and Item2 can be defined as:

Item1 := λt.t True
Item2 := λt.t False

To implement tuple in C#, its function type needs to be identified. The tuple function accepts argument f, which is either function True of function False, so f is of function type Boolean. In the body of tuple function, f is applied, and f returns dynamic. So tuple virtually is of function type Boolean -> dynamic:

using static ChurchBoolean;

// Tuple is the alias of (dynamic -> dynamic -> dynamic) -> dynamic.
public delegate dynamic Tuple<out T1, out T2>(Boolean f);

public static partial class ChurchTuple<T1, T2>
{
    public static readonly Func<T1, Func<T2, Tuple<T1, T2>>> 
        Create = item1 => item2 => f => f(item1)(item2);

    // Item1 = tuple => tuple(True)
    public static readonly Func<Tuple<T1, T2>, T1> 
        Item1 = tuple => (T1)(object)tuple(True);

    // Item2 = tuple => tuple(False)
    public static readonly Func<Tuple<T1, T2>, T2> 
        Item2 = tuple => (T2)(object)tuple(False);
}

There are type conversions in Item1/Item2 functions. At compiled time, the tuple function returns dynamic, and at runtime it actually it calls True/False function to return either item1 or item2 . So the type conversions are always safe. Also notice here tuple function’s return value cannot be directly converted to T1 or T2, because of a bug of C# runtime binding layer. The workaround is to convert dynamic to object first, then convert to T1 or T2.

The following are the extension methods for convenience:

public static partial class TupleExtensions
{
    public static T1 Item1<T1, T2>(this Tuple<T1, T2> tuple) => ChurchTuple<T1, T2>.Item1(tuple);

    public static T2 Item2<T1, T2>(this Tuple<T1, T2> tuple) => ChurchTuple<T1, T2>.Item2(tuple);
}

For example, a point can be a tuple of 2 numerals:

internal static void Point(Numeral x, Numeral y)
{
    Tuple<Numeral, Numeral> point1 = ChurchTuple<Numeral, Numeral>.Create(x)(y);
    Numeral x1 = point1.Item1();
    Numeral y1 = point1.Item1();

    // Move up.
    Numeral y2 = y1.Increase();
    Tuple<Numeral, Numeral> point2 = ChurchTuple<Numeral, Numeral>.Create(x1)(y2);
}

Tuple operators

The Swap function accepts a tuple (x, y), swaps its first item and second item, and returns a new tuple (y, x):

Swap := λt.CreateTuple (Item2 t)(Item1 t)

Apparently, Swap is of function type Tuple<T1, T2> -> Tuple<T2, T1>:

// Swap = tuple => Create(tuple.Item2())(tuple.Item1())
public static readonly Func<Tuple<T1, T2>, Tuple<T2, T1>>
    Swap = tuple => ChurchTuple<T2, T1>.Create(tuple.Item2())(tuple.Item1());

The Shift function accepts a tuple (x, y) and a function f, and returns a new tuple (y, f y):

Shift := λf.λt.CreateTuple (Item2 t) (f (Item2 t))

Here assume the argument tuple (x, y) is of type Tuple<T1, T2>, regarding f is applied with y, assume f returns TResult type, then f is of function type T2 -> TResult, so that the returned new tuple (y, f y) is of type Tuple<T2, TResult>. As a result, Shift is of type Tuple<T1, T2> -> (T2 -> TResult) -> Tuple<T2, TResult>:

public static partial class ChurchTuple<T1, T2, TResult>
{
    // Shift = f => tuple => Create(tuple.Item2())(f(tuple.Item1()))
    public static readonly Func<Func<T2, TResult>, Func<Tuple<T1, T2>, Tuple<T2, TResult>>>
        Shift = f => tuple => ChurchTuple<T2, TResult>.Create(tuple.Item2())(f(tuple.Item2()));
}

And their extension methods:

public static Tuple<T2, T1> Swap<T1, T2>(this Tuple<T1, T2> tuple) => ChurchTuple<T1, T2>.Swap(tuple);

public static Tuple<T2, TResult> Shift<T1, T2, TResult>(this Tuple<T1, T2> tuple, Func<T2, TResult> f) => 
    ChurchTuple<T1, T2, TResult>.Shift(f)(tuple);

Here Shift function can be used to define the Subtract function for Church numerals. Remember a Church numeral n can be viewed as to apply Increase n times from 0:

n Increase 0
≡ n

Applying Shift with Increase and a tuple of Church numerals, it returns a new tuple of Church numerals, so this application can repeat forever:

Shift Increase (0, 0)
≡ (0, Increase 0)
≡ (0, 1)

  Shift Increase (0, 1)
≡ (1, Increase 1)
≡ (1, 2)

  Shift Increase (1, 2)
≡ (2, Increase 2)
≡ (2, 3)

...

In another word, partially applying Shift with Increase is a function that can be repeatedly applied with a tuple of Church numerals:

(Shift Increase) (0, 0)                                       ≡ (Shift Increase)1 (0, 0) ≡ 1 (Shift Increase) (0, 0) 
≡ (0, 1)

  (Shift Increase) (0, 1)
≡ (Shift Increase) ((Shift Increase) (0, 0))
≡ (Shift Increase) ∘ (Shift Increase) (0, 0)                    ≡ (Shift Increase)2 (0, 0) ≡ 2 (Shift Increase) (0, 0) 
≡ (1, 2)

  (Shift Increase) (1, 2)
≡ (Shift Increase) ((Shift Increase) ∘ (Shift Increase) (0, 0))
≡ (Shift Increase) ∘ (Shift Increase) ∘ (Shift Increase) (0, 0) ≡ (Shift Increase)3 (0, 0) ≡ 3 (Shift Increase) (0, 0) 
≡ (2, 3)

...

So generally:

n (Shift Increase) (0, 0)
≡ (n - 1, n)

As a result, to decrease n to n – 1, just apply n with function (Shift Increase) and tuple (0, 0), get the result tuple (n – 1, n), and return its first item:

Item1 (n (Shift Increase) (0, 0))
≡ Item1 (n - 1, n)
≡ n - 1

So Decrease can be defined as:

Decrease := λn.Item1 (n (Shift Increase) (CreateTuple 0 0))

And C#:

// Decrease = n => n(tuple => tuple.Shift(Increase))(0, 0).Item1();
public static readonly Func<Numeral, Numeral> Decrease = n =>
    ((Tuple<Numeral, Numeral>)n
        (tuple => ((Tuple<Numeral, Numeral>)tuple).Shift(Increase))
        (ChurchTuple<Numeral, Numeral>.Create(Zero)(Zero)))
    .Item1();

N-tuple

An easy way is to model n-tuple as a 2-tuple of the first value and a (n-1)-tuple of the rest values. A 3 tuple of values 1, 2, 3 can be represented by nested 2-tuples as (a, (b, c)), a 4-tuple of values 1, 2, 3, 4 can be represented by nested 2-tuples (1, (2, (3, 4))), etc., and a n tuple of values 1, 2, 3, …, n can be represented by nested 2 tuples (1, (2, (3, (…(n-1, n)…)))). For example, the following is the definition of 3 tuple:

Create3Tuple := λx.λy.λz.CreateTuple x (CreateTuple y z)

3TupleItem1 := λt.Item1 t
3TupleItem2 := λt.Item1 (Item2 t)
3TupleItem3 := λt.Item2 (Item2 t)

And in C#:

public delegate dynamic Tuple<out T1, out T2, out T3>(Boolean f);

public static partial class ChurchTuple<T1, T2, T3>
{
    // Create = item1 => item2 => item3 => Create(item1)(Create(item2)(item3))
    public static readonly Func<T1, Func<T2, Func<T3, Tuple<T1, T2, T3>>>>
        Create = item1 => item2 => item3 => new Tuple<T1, T2, T3>(ChurchTuple<T1, Tuple<T2, T3>>.Create(item1)(ChurchTuple<T2, T3>.Create(item2)(item3)));

    // Item1 = tuple.Item1()
    public static readonly Func<Tuple<T1, T2, T3>, T1>
        Item1 = tuple => new Tuple<T1, Tuple<T2, T3>>(tuple).Item1();

    // Item2 = tuple.Item2().Item1()
    public static readonly Func<Tuple<T1, T2, T3>, T2>
        Item2 = tuple => new Tuple<T1, Tuple<T2, T3>>(tuple).Item2().Item1();

    // Item3 = tuple.Item2().Item2()
    public static readonly Func<Tuple<T1, T2, T3>, T3>
        Item3 = tuple => new Tuple<T1, Tuple<T2, T3>>(tuple).Item2().Item2();
}

public static partial class TupleExtensions
{
    public static T1 Item1<T1, T2, T3>(this Tuple<T1, T2, T3> tuple) => ChurchTuple<T1, T2, T3>.Item1(tuple);

    public static T2 Item2<T1, T2, T3>(this Tuple<T1, T2, T3> tuple) => ChurchTuple<T1, T2, T3>.Item2(tuple);

    public static T3 Item3<T1, T2, T3>(this Tuple<T1, T2, T3> tuple) => ChurchTuple<T1, T2, T3>.Item3(tuple);
}

Signed numeral

With tuple, a signed numeral (integer) can be modeled by a pair of Church numerals (natural numbers), where the first item represents the positive value, and the second item represents the negative value:

SignedNumeral := Tuple

For example (1, 0) and (2, 1) models 1, (0, 2) and (1, 3) models –2, (0, 0) and (1, 1) models 0, etc.:

1 := (1, 0) ≡ (2, 1) ≡ (3, 2) ≡ (4, 3) ≡ ...
 0 := (0, 0) ≡ (1, 1) ≡ (2, 2) ≡ (3, 3) ≡ ...
-2 := (0, 2) ≡ (1, 3) ≡ (2, 4) ≡ (3, 5) ≡ ...

In C#, the function type SignedNumeral is the same as Tuple, except SignedNumeral is not open generic type:

// SignedNumeral is the alias of Tuple<Numeral, Numeral>.
public delegate dynamic SignedNumeral(Boolean f);

Church numeral represents natural number. So Converting a Church numeral n to signed number is easy, just make it a tuple (n, 0):

Sign := λn.CreateTuple n 0

To negate a signed numeral, just swap its positive value and negative value:

Negate := Swap

And it is straightforward to get the positive value and the negative value from a signed number:

Positive := Item1
Negative := Item2

Signed numeral like (4, 3), (3, 3), (3, 5) can be formatted to have at least one 0: (1, 0), (0, 0), (0, 2). For a signed number s represented by (p, n), If p >= n, then it is (p - n, 0), otherwise it is (0, n – p):

Format := λs.If (sp >=  sn) (λx.(sp - sn, 0)) (λx.(0, sn - sp))

Here Sp is the positive value of s, and sn the the negative value of s.

The following are these function’s C# implementation, and the extension methods:

using static ChurchBoolean;
using static ChurchNumeral;

public static partial class ChurchSignedNumeral
{
    // Sign = n => (n, 0)
    public static readonly Func<Numeral, SignedNumeral>
        Sign = n => new SignedNumeral(ChurchTuple<Numeral, Numeral>.Create(n)(Zero));

    // Negate = signed => signed.Swap()
    public static readonly Func<SignedNumeral, SignedNumeral>
        Negate = signed => new SignedNumeral(new Tuple<Numeral, Numeral>(signed).Swap());

    // Positive = signed => signed.Item1()
    public static readonly Func<SignedNumeral, Numeral>
        Positive = signed => new Tuple<Numeral, Numeral>(signed).Item1();

    // Negative = signed => signed.Item2()
    public static readonly Func<SignedNumeral, Numeral>
        Negative = signed => new Tuple<Numeral, Numeral>(signed).Item2();

    // Format = signed =>
    //    If(positive >= negative)
    //        (_ => (positive - negative, 0))
    //        (_ => (0, negative - positive))
    public static readonly Func<SignedNumeral, SignedNumeral>
        Format = signed =>
            If(signed.Positive().IsGreaterThanOrEqualTo(signed.Negative()))
                (_ => signed.Positive().Subtract(signed.Negative()).Sign())
                (_ => signed.Negative().Subtract(signed.Positive()).Sign().Negate());
}

public static partial class SignedNumeralExtensions
{
    public static SignedNumeral Sign(this Numeral n) => ChurchSignedNumeral.Sign(n);

    public static SignedNumeral Negate(this SignedNumeral signed) => ChurchSignedNumeral.Negate(signed);

    public static Numeral Positive(this SignedNumeral signed) => ChurchSignedNumeral.Positive(signed);

    public static Numeral Negative(this SignedNumeral signed) => ChurchSignedNumeral.Negative(signed);

    public static SignedNumeral Format(this SignedNumeral signed) => ChurchSignedNumeral.Format(signed);
}

Arithmetic operators

Naturally, for signed numbers a, b:

a + b
≡ (ap, an) + (bp, bn)
≡ (ap - an) + (bp - bn)
≡ (ap + bp, an + bn)

  a - b
≡ (ap, an) - (bp, bn)
≡ (ap - an) - (bp - bn)
≡ (ap + bn, an + bp)

  a * b
≡ (ap, an) * (bp, bn)
≡ (ap - an) * (bp - bn)
≡ (ap * bp + an * bn, ap * bn + an * bp)

  a / b
≡ (ap, an) / (bp, bn)
≡ (ap - an) / (bp - bn)
≡ (ap / bp + an / bn, ap / bn + an / bp)

So in lambda calculus:

AddSigned := λa.λb.Format (CreateTuple (ap + bp) (an + bn))
SubtractSigned := λa.λb.Format (CreateTuple (ap + bn) (an + bp))
MultiplySigned := λa.λb.Format (CreateTuple (ap * bp + an * bn) (ap * bn + an * bp))

Division is more tricky because a and b’s positive and negative values can be 0. In this case, just return 0 when dividing by 0:

DivideByIgnoreZero := λa.λb.If (IsZero b) (λx.0) (λx.DivideBy a b)

Here the DivideBy function for Church numeral is used. As fore mentioned, this DivideBy function is not well defined. It is temporarily used here and will be revisited later. So division can be defined as:

DivideBySigned := λa.λb.Format (CreateTuple ((DivideByIgnoreZero ap bp) + (DivideByIgnoreZero an bn)) ((DivideByIgnoreZero ap bn) + (DivideByIgnoreZero an bp)))

The following are the C# implementations and the extension methods:

public static partial class ChurchSignedNumeral
{
    // Add = a => b => (a.Positive() + b.Positive(), a.Negative() + b.Negative()).Format()
    public static readonly Func<SignedNumeral, Func<SignedNumeral, SignedNumeral>>
        Add = a => b => new SignedNumeral(ChurchTuple<Numeral, Numeral>.Create
                (a.Positive().Add(b.Positive()))
                (a.Negative().Add(b.Negative())))
            .Format();

    // Subtract = a => b => (a.Positive() + b.Negative(), a.Negative() + b.Positive()).Format()
    public static readonly Func<SignedNumeral, Func<SignedNumeral, SignedNumeral>>
        Subtract = a => b => new SignedNumeral(ChurchTuple<Numeral, Numeral>.Create
                (a.Positive().Add(b.Negative()))
                (a.Negative().Add(b.Positive())))
            .Format();

    // Multiply = a => b => (a.Positive() * b.Positive() + a.Negative() * b.Negative(), a.Positive() * b.Negative() + a.Negative() * b.Positive()).Format()
    public static readonly Func<SignedNumeral, Func<SignedNumeral, SignedNumeral>>
        Multiply = a => b => new SignedNumeral(ChurchTuple<Numeral, Numeral>.Create
                (a.Positive().Multiply(b.Positive()).Add(a.Negative().Multiply(b.Negative())))
                (a.Positive().Multiply(b.Negative()).Add(a.Negative().Multiply(b.Positive()))))
            .Format();

    // / = dividend => divisor => If(divisor.IsZero())(_ => 0)(_ => dividend.DivideBy(divisor))
    private static readonly Func<Numeral, Func<Numeral, Numeral>> 
        DivideByIgnoreZero = dividend => divisor =>
            ChurchBoolean<Numeral>.If(divisor.IsZero())
                (_ => Zero)
                (_ => dividend.DivideBy(divisor));

    // DivideBy = dividend => divisor => (dividend.Positive() / divisor.Positive() + dividend.Negative() / divisor.Negative(), dividend.Positive() / divisor.Negative() + dividend.Negative() / divisor.Positive()).Format();
    public static readonly Func<SignedNumeral, Func<SignedNumeral, SignedNumeral>>
        DivideBy = dividend => divisor => new SignedNumeral(ChurchTuple<Numeral, Numeral>.Create
                (DivideByIgnoreZero(dividend.Positive())(divisor.Positive()).Add(DivideByIgnoreZero(dividend.Negative())(divisor.Negative())))
                (DivideByIgnoreZero(dividend.Positive())(divisor.Negative()).Add(DivideByIgnoreZero(dividend.Negative())(divisor.Positive()))))
            .Format();
}

public static partial class SignedNumeralExtensions
{
    public static SignedNumeral Add(this SignedNumeral a, SignedNumeral b) => ChurchSignedNumeral.Add(a)(b);

    public static SignedNumeral Subtract(this SignedNumeral a, SignedNumeral b) => ChurchSignedNumeral.Subtract(a)(b);

    public static SignedNumeral Multiply(this SignedNumeral a, SignedNumeral b) => ChurchSignedNumeral.Multiply(a)(b);

    public static SignedNumeral DivideBy(this SignedNumeral dividend, SignedNumeral divisor) => ChurchSignedNumeral.DivideBy(dividend)(divisor);
}

And the following code demonstrate how these operators work:

[TestClass]
public class ChurchSignedNumeralTests
{
    [TestMethod]
    public void SignNegatePositiveNegativeTest()
    {
        SignedNumeral signed = 0U.Church().Sign();
        Assert.IsTrue(0U == signed.Positive().Unchurch());
        Assert.IsTrue(0U == signed.Negative().Unchurch());
        signed = signed.Negate();
        Assert.IsTrue(0U == signed.Positive().Unchurch());
        Assert.IsTrue(0U == signed.Negative().Unchurch());

        signed = 1U.Church().Sign();
        Assert.IsTrue(1U == signed.Positive().Unchurch());
        Assert.IsTrue(0U == signed.Negative().Unchurch());
        signed = signed.Negate();
        Assert.IsTrue(0U == signed.Positive().Unchurch());
        Assert.IsTrue(1U == signed.Negative().Unchurch());

        signed = 2U.Church().Sign();
        Assert.IsTrue(2U == signed.Positive().Unchurch());
        Assert.IsTrue(0U == signed.Negative().Unchurch());
        signed = signed.Negate();
        Assert.IsTrue(0U == signed.Positive().Unchurch());
        Assert.IsTrue(2U == signed.Negative().Unchurch());

        signed = 123U.Church().Sign();
        Assert.IsTrue(123U == signed.Positive().Unchurch());
        Assert.IsTrue(0U == signed.Negative().Unchurch());
        signed = signed.Negate();
        Assert.IsTrue(0U == signed.Positive().Unchurch());
        Assert.IsTrue(123U == signed.Negative().Unchurch());

        signed = new SignedNumeral(ChurchTuple<Numeral, Numeral>.Create(12U.Church())(23U.Church()));
        Assert.IsTrue(12U == signed.Positive().Unchurch());
        Assert.IsTrue(23U == signed.Negative().Unchurch());
        signed = signed.Negate();
        Assert.IsTrue(23U == signed.Positive().Unchurch());
        Assert.IsTrue(12U == signed.Negative().Unchurch());
    }

    [TestMethod]
    public void FormatWithZeroTest()
    {
        SignedNumeral signed = new SignedNumeral(ChurchTuple<Numeral, Numeral>.Create(12U.Church())(23U.Church()));
        signed = signed.Format();
        Assert.IsTrue(0U == signed.Positive().Unchurch());
        Assert.IsTrue(11U == signed.Negative().Unchurch());

        signed = new SignedNumeral(ChurchTuple<Numeral, Numeral>.Create(23U.Church())(12U.Church()));
        signed = signed.Format();
        Assert.IsTrue(11U == signed.Positive().Unchurch());
        Assert.IsTrue(0U == signed.Negative().Unchurch());
    }

    [TestMethod]
    public void AddTest()
    {
        SignedNumeral a = 0U.Church().Sign();
        SignedNumeral b = 0U.Church().Sign();
        SignedNumeral result = a.Add(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(0U == result.Negative().Unchurch());

        a = 1U.Church().Sign();
        b = 1U.Church().Sign().Negate();
        result = a.Add(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(0U == result.Negative().Unchurch());

        a = 3U.Church().Sign();
        b = 5U.Church().Sign().Negate();
        result = a.Add(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(2U == result.Negative().Unchurch());
    }

    [TestMethod]
    public void SubtractTest()
    {
        SignedNumeral a = 0U.Church().Sign();
        SignedNumeral b = 0U.Church().Sign();
        SignedNumeral result = a.Subtract(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(0U == result.Negative().Unchurch());

        a = 1U.Church().Sign();
        b = 1U.Church().Sign().Negate();
        result = a.Subtract(b);
        Assert.IsTrue(2U == result.Positive().Unchurch());
        Assert.IsTrue(0U == result.Negative().Unchurch());

        a = 3U.Church().Sign();
        b = 5U.Church().Sign().Negate();
        result = a.Subtract(b);
        Assert.IsTrue(8U == result.Positive().Unchurch());
        Assert.IsTrue(0U == result.Negative().Unchurch());
    }

    [TestMethod]
    public void MultiplyTest()
    {
        SignedNumeral a = 0U.Church().Sign();
        SignedNumeral b = 0U.Church().Sign();
        SignedNumeral result = a.Multiply(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(0U == result.Negative().Unchurch());

        a = 1U.Church().Sign();
        b = 1U.Church().Sign().Negate();
        result = a.Multiply(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(1U == result.Negative().Unchurch());

        a = 3U.Church().Sign();
        b = 5U.Church().Sign().Negate();
        result = a.Multiply(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(15U == result.Negative().Unchurch());
    }

    [TestMethod]
    public void DivideByTest()
    {
        SignedNumeral a = 0U.Church().Sign();
        SignedNumeral b = 0U.Church().Sign();
        SignedNumeral result = a.DivideBy(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(0U == result.Negative().Unchurch());

        a = 1U.Church().Sign();
        b = 1U.Church().Sign().Negate();
        result = a.DivideBy(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(1U == result.Negative().Unchurch());

        a = 11U.Church().Sign();
        b = 5U.Church().Sign().Negate();
        result = a.DivideBy(b);
        Assert.IsTrue(0U == result.Positive().Unchurch());
        Assert.IsTrue(2U == result.Negative().Unchurch());
    }
}

Lambda Calculus via C# (3) Numeral, Arithmetic and Predicate

Thu, 07 Nov 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Lambda Calculus via C# series]

Anonymous functions can also model numerals and their arithmetic. In Church encoding, a natural number n is represented by a function that calls a given function for n times. This representation is called Church Numeral.

Church numerals

Church numerals are defined as:

0 := λfx.x                  ≡ λf.λx.x
1 := λfx.f x                ≡ λf.λx.f x
2 := λfx.f (f x)            ≡ λf.λx.f (f x)
3 := λfx.f (f (f x))        ≡ λf.λx.f (f (f x))
...
n := λfx.f (f ... (f x)...) ≡ λf.λx.f (f .u.. (f x)...)

So a Church numeral n is a higher order function, it accepts a function f and an argument x. When n is applied, it repeatedly applies f for n times by starting with x, and returns the result. If n is 0, f is not applied (in another word, f is applied 0 times), and x is directly returned.

0 f x ≡ x
1 f x ≡ f x
2 f x ≡ f (f x)
3 f x ≡ f (f (f x))
...
n f x ≡ f (f (... (f x)...))

According to the definition of function composition:

f (f x) ≡ (f ∘ f) x

This definition is equivalent to compose f for n time:

0 := λfx.x                  ≡ λf.λx.x                   ≡ λf.λx.f0 x
1 := λfx.f x                ≡ λf.λx.f x                 ≡ λf.λx.f1 x
2 := λfx.f (f x)            ≡ λf.λx.(f ∘ f) x           ≡ λf.λx.f2 x
3 := λfx.f (f (f x))        ≡ λf.λx.(f ∘ f ∘ f) x       ≡ λf.λx.f3 x
...
n := λfx.f (f ... (f x)...) ≡ λf.λx.(f ∘ f ∘ ... ∘ f) x ≡ λf.λx.fn x

The partial application with f is the composition of f, so Church numeral n can be simply read as – do something n times:

0 f ≡ f0
1 f ≡ f1
2 f ≡ f2
3 f ≡ f3
...
n f ≡ fn

In C#, x can be anything, so leave its type as dynamic. f can be viewed as a function accept a value x and returns something, and f can also accept its returned value again, so f is of type dynamic -> dynamic. And n’ return type is the same as f’s return type, so n returns dynamic too. As a result, n can be virtually viewed as curried function type (dynamic -> dynamic) –> dynamic -> dynamic, which in C# is represented by Func<Func<dynamic, dynamic>, Func<dynamic, dynamic>>. Similar to the C# implementation of Church Boolean, a alias Numeral can be defined:

// Curried from (dynamic -> dynamic, dynamic) -> dynamic.
// Numeral is the alias of (dynamic -> dynamic) -> dynamic -> dynamic.
public delegate Func<dynamic, dynamic> Numeral(Func<dynamic, dynamic> f);

Based on the definition:

public static partial class ChurchNumeral
{
    public static readonly Numeral
        Zero = f => x => x;

    public static readonly Numeral
        One = f => x => f(x);

    public static readonly Numeral
        Two = f => x => f(f(x));

    public static readonly Numeral
        Three = f => x => f(f(f(x)));

    // ...
}

Also since n f ≡ fn, n can be also implemented with composition of f:

public static readonly Numeral
    OneWithComposition = f => f;

// Two = f => f o f
public static readonly Numeral
    TwoWithComposition = f => f.o(f);

// Three = f => f o f o f
public static readonly Numeral
    ThreeWithComposition = f => f.o(f).o(f);

// ...

Here the o operator is the forward composition extension method defined previously. Actually, instead of defining each number individually, Church numeral can be defined recursively by increase or decrease.

Increase and decrease

By observing the definition and code, there are some patterns when the Church numeral increases from 0 to 3. In the definitions of Church numerals:

0 := λf.λx.x
1 := λf.λx.f (x)
2 := λf.λx.f (f x)
3 := λf.λx.f (f (f x))
...

The expressions in the parenthesis can be reduced from the following function applications expressions:

0 f x ≡ x
1 f x ≡ f x
2 f x ≡ f (f x)
...

With substitution, Church numerals’ definition become:

0 := λf.λx.x
1 := λf.λx.f (0 f x)
2 := λf.λx.f (1 f x)
3 := λf.λx.f (2 f x)
...

This shows how the the Church numerals increases. Generally, given a Church numeral n, the next numeral n + 1 is λf.λx.f (n f x). So:

Increase := λn.λf.λx.f (n f x)

In C#, this is:

public static Func<Numeral, Numeral> 
    Increase = n => f => x => f(n(f)(x));

In the other way, Church numeral n is to compose f for n times:

n f ≡ fn

So increasing n means to compose f for one more time:

Increase := λn.λf.f ∘ fn ≡ λn.λf.f ∘ (n f)

And in C#:

public static readonly Func<Numeral, Numeral> 
    IncreaseWithComposition = n => f => f.o(n(f));

To decrease a Church numeral n, when n is 0, the result is defined as 0, when n is positive, the result is n – 1. The Decrease function is more complex:

Decrease := λn.λf.λx.n (λg.λh.h (g f)) (λv.x) Id

When n is 0, regarding n f ≡ fn, applying Decrease with 0 can be reduced as:

Decrease 0
≡ λf.λx.0 (λg.λh.h (g f)) (λv.x) Id
≡ λf.λx.(λg.λh.h (g f))0 (λv.x) Id
≡ λf.λx.(λv.x) Id
≡ λf.λx.x
≡ λf.λx.f0 x

The last expression is the definition of 0.

When n is positive, regarding function function composition is associative, the expression n (λg.λh.h (g f)) (λu.x) can be reduced first. When n is 1, 2, 3, ...:

1 (λg.λh.h (g f)) (λv.x)
≡ (λg.λh.h (g f))1 (λv.x)
≡ (λg.λh.h (g f)) (λv.x)
≡ λh.h ((λv.x) f)
≡ λh.h x
≡ λh.h (f0 x) 

  2 (λg.λh.h (g f)) (λv.x)
≡ (λg.λh.h (g f))2 (λv.x)
≡ (λg.λh.h (g f)) ∘ (λg.λh.h (g f))1 (λv.x)
≡ (λg.λh.h (g f)) (λh.h (f0 x))
≡ λh.h (λh.h (f0 x) f)
≡ λh.h (f (f0 x))
≡ λh.h (f1 x)

  3 (λg.λh.h (g f)) (λv.x)
≡ (λg.λh.h (g f))3 (λv.x)
≡ (λg.λh.h (g f)) ∘ (λg.λh.h (g f))2 (λv.x)
≡ (λg.λh.h (g f)) (λh.h (f1 x))
≡ λh.h ((λh.h (f1 x)) f)
≡ λh.h (f (f1 x))
≡ λh.h (f2 x)

...

And generally:

n (λg.λh.h (g f)) (λv.x)
≡ λh.h (fn - 1 x)

So when Decrease is applied with positive n:

Decrease n
≡ λf.λx.n (λg.λh.h (g f)) (λv.x) Id
≡ λf.λx.(λh.h (fn - 1 x)) Id
≡ λf.λx.Id (fn - 1 x)
≡ λf.λx.fn - 1 x

The returned result is the definition of n – 1. In the following C# implementation, a lot of noise of type information is involved to implement complex lambda expression:

// Decrease = n => f => x => n(g => h => h(g(f)))(_ => x)(Id)
public static readonly Func<Numeral, Numeral> 
    Decrease = n => f => x => n(g => new Func<Func<dynamic, dynamic>, dynamic>(h => h(g(f))))(new Func<Func<dynamic, dynamic>, dynamic>(_ => x))(new Func<dynamic, dynamic>(Functions<dynamic>.Id));

Here are the actual types of the elements in above lambda expression at runtime:

g: (dynamic -> dynamic) -> dynamic
h: dynamic -> dynamic
g(f): dynamic
h(g(f)): dynamic
h => h(g(f)): (dynamic -> dynamic) -> dynamic
g => h => h(g(f)): ((dynamic -> dynamic) -> dynamic) -> (dynamic -> dynamic) -> dynamic
n(g => h => h(g(f))): ((dynamic -> dynamic) -> dynamic) -> (dynamic -> dynamic) -> dynamic
_ => x: (dynamic -> dynamic) -> dynamic
n(g => h => h(g(f)))(_ => x): (dynamic -> dynamic) -> dynamic
Id: dynamic -> dynamic
n(g => h => h(g(f)))(_ => x)(Id): dynamic

At compile time, function types must be provided for a few elements. When n is applied, C# compiler expects its first argument g => h => h(g(f)) to be of type dynamic => dynamic. So C# compiler infers g to dynamic, but cannot infer the type of h => h(g(f)), which can be expression tree or anonymous function, so the constructor call syntax is used here to specify it is a function of type (dynamic -> dynamic) -> dynamic. Similarly, C# compiler expects n’s second argument to be dynamic, and C# compiler cannot infer the type of _ => x, so the constructor syntax is used again for _ => x. Also, Functions<dynamic>.Id is of Unit<dynamic> type, while at runtime a dynamic -> dynamic function is expected. Unit<dynamic> is alias of function type dynamic –> dynamic, but the conversion does not happen automatically at runtime, so the constructor syntax is used once again to indicate the function type conversion.

Later after introducing Church pair, a cleaner version of Decrease will be implemented.

Arithmetic operators

To implement add operation, according to the definition, Church numeral a adding Church numeral b means to apply f for a times, then apply f again for b times:

Add := λa.λb.λf.λx.b f (a f x)

With the definition of function composition, Add can be also defined as:

Add := λa.λb.λf.fa ∘ fb ≡ λa.λb.λf.(a f) ∘ (b f)

So in C#:

public static readonly Func<Numeral, Func<Numeral, Numeral>>  
    Add = a => b => f => x => b(f)(a(f)(x));

public static readonly Func<Numeral, Func<Numeral, Numeral>> 
    AddWithComposition = a => b => f => a(f).o(b(f));

With Increase function, Add can also be defined as increase a for b times:

Add := λa.λb.b Increase a

In C#, there are some noise of type information again:

public static readonly Func<Numeral, Func<Numeral, Numeral>>
    AddWithIncrease = a => b => b(Increase)(a);

Unfortunately, the above code cannot be compiled, because b is a function of type (dynamic -> dynamic) -> dynamic x -> dynamic. So its first argument f must be a function of type dynamic -> dynamic. Here, Increase is of type Numeral -> Numeral, and b(Increase) cannot be compiled. The solution is to eta convert Increase to a wrapper function λn.Increase n:

Add := λa.λb.a (λn.Increase n) b

So that in C#:

// Add = a => b => b(Increase)(a)
// η conversion:
// Add = a => b => b(n => Increase(n))(a)
public static readonly Func<Numeral, Func<Numeral, Numeral>>
    AddWithIncrease = a => b => b(n => Increase(n))(a);

Since a dynamic -> dynamic function is expected and the wrapper function n => Increase(n), n inferred to be of type dynamic. Increase(n) still returns Numeral, so the wrapper function is of type dynamic -> Numeral. Regarding dynamic is just object, and Numeral derives from object, with support covariance in C#, the wrapper function is implicitly converted to dynamic -> dynamic, so calling b with the wrapper function can be compiled.

Similarly, Church numeral a subtracting b can be defined as decrease a for b times, a multiplying b can be defined as adding a for b times to 0, and raising a to the power b can be defined as multiplying a for n times with 1:

Subtract := λa.λb.b Decrease a
Multiply := λa.λb.b (Add a) 0
Power := λa.λb.b (Multiply a) 1

The C# implementation are in the same pattern:

// Subtract = a => b => b(Decrease)(a)
// η conversion:
// Subtract = a => b => b(n => Decrease(n))(a)
public static readonly Func<Numeral, Func<Numeral, Numeral>>
    Subtract = a => b => b(n => Decrease(n))(a);

// Multiply = a => b => b(Add(a))(a)
// η conversion:
// Multiply = a => b => b(n => Add(a)(n))(Zero)
public static readonly Func<Numeral, Func<Numeral, Numeral>>
    Multiply = a => b => b(n => Add(a)(n))(Zero);

// Pow = a => b => b(Multiply(a))(a)
// η conversion:
// Pow = a => b => b(n => Multiply(a)(n))(1)
public static readonly Func<Numeral, Func<Numeral, Numeral>>
    Pow = a => b => b(n => Multiply(a)(n))(One);

Similar to Church Boolean operators, the above arithmetic operators can also be wrapped as extension method for convenience:

public static partial class NumeralExtensions
{
    public static Numeral Increase(this Numeral n) => ChurchNumeral.Increase(n);

    public static Numeral Decrease(this Numeral n) => ChurchNumeral.Decrease(n);

    public static Numeral Add(this Numeral a, Numeral b) => ChurchNumeral.Add(a)(b);

    public static Numeral Subtract(this Numeral a, Numeral b) => ChurchNumeral.Subtract(a)(b);

    public static Numeral Multiply(this Numeral a, Numeral b) => ChurchNumeral.Multiply(a)(b);

    public static Numeral Pow(this Numeral mantissa, Numeral exponent) => ChurchNumeral.Pow(mantissa)(exponent);
}

Predicate and relational operators

Predicate is function returning Church Boolean. For example, the following function predicate whether a Church numeral n is 0:

IsZero := λn.n (λx.False) True

When n is 0, (λx.False) is not applied, and IsZero directly returns True. When n is positive, (λx.False) is applied for n times. (λx.False) always return False, so IsZero returns False. The following are the implementation and extension method:

public static partial class ChurchPredicate
{
    public static readonly Func<Numeral, Boolean> 
        IsZero = n => n(_ => False)(True);
}

public static partial class NumeralExtensions
{
    public static Boolean IsZero(this Numeral n) => ChurchPredicate.IsZero(n);
}

With IsZero, it is easy to define functions to compare 2 Church numerals a and b. According the to definition of Decrease and Subtract, when a – b is 0, a is either equal to b, or less than b. So IsLessThanOrEqualTo can be defined with IsZero and Subtract:

IsLessThanOrEqualTo := λa.λb.IsZero (Subtract a b)

IsGreaterThanOrEqualTo is similar:

IsGreaterThanOrEqualTo := λa.λb.IsZero (Subtract b a)

Then these 2 functions can define IsEqualTo:

IsEqualTo := λa.λb.And (IsLessThanOrEqualTo a b) (IsGreaterThanOrEqualTo a b)

The opposite of these functions are IsGreaterThan, IsLessThan, IsNotEqual. They can be defined with Not:

IsGreaterThan := λa.λb.Not (IsLessThanOrEqualTo a b)
IsLessThan := λa.λb.Not (IsGreaterThanOrEqualTo a b)
IsNotEqualTo := λa.λb.Not (IsEqualTo a b)

The following are the C# implementation of these 6 predicates:

public static partial class ChurchPredicate
{
    public static readonly Func<Numeral, Func<Numeral, Boolean>> 
        IsLessThanOrEqualTo = a => b => a.Subtract(b).IsZero();

    public static readonly Func<Numeral, Func<Numeral, Boolean>> 
        IsGreaterThanOrEqualTo = a => b => b.Subtract(a).IsZero();

    public static readonly Func<Numeral, Func<Numeral, Boolean>>
        IsEqualTo = a => b => IsLessThanOrEqualTo(a)(b).And(IsGreaterThanOrEqualTo(a)(b));

    public static readonly Func<Numeral, Func<Numeral, Boolean>>
        IsGreaterThan = a => b => IsLessThanOrEqualTo(a)(b).Not();

    public static readonly Func<Numeral, Func<Numeral, Boolean>> 
        IsLessThan = a => b => IsGreaterThanOrEqualTo(a)(b).Not();

    public static readonly Func<Numeral, Func<Numeral, Boolean>>
        IsNotEqualTo = a => b => IsEqualTo(a)(b).Not();
}

public static partial class NumeralExtensions
{
    public static Boolean IsLessThanOrEqualTo(this Numeral a, Numeral b) => ChurchPredicate.IsLessThanOrEqualTo(a)(b);

    public static Boolean IsGreaterThanOrEqualTo(this Numeral a, Numeral b) => ChurchPredicate.IsGreaterThanOrEqualTo(a)(b);

    public static Boolean IsEqualTo(this Numeral a, Numeral b) => ChurchPredicate.IsEqualTo(a)(b);

    public static Boolean IsGreaterThan(this Numeral a, Numeral b) => ChurchPredicate.IsGreaterThan(a)(b);

    public static Boolean IsLessThan(this Numeral a, Numeral b) => ChurchPredicate.IsLessThan(a)(b);

    public static Boolean IsNotEqualTo(this Numeral a, Numeral b) => ChurchPredicate.IsNotEqualTo(a)(b);
}

Attempt of recursion

The division of natural numbers can be defined with arithmetic and relation operators:

a / b := if a >= b then 1 + (a – b) / b else 0

This is a recursive definition. If defining division in this way lambda calculus, the function name is referred in its own body:

DivideBy := λa.λb.If (IsGreaterThanOrEqualTo a b) (λx.Add One (DivideBy (Subtract a b) b)) (λx.Zero)

As fore mentioned, in lambda calculus, functions are anonymously by default, and names are just for readability. Here the self reference does not work with anonymous function:

λa.λb.If (IsGreaterThanOrEqualTo a b) (λx.Add One (? (Subtract a b) b)) (λx.Zero)

So the above DivideBy function definition is illegal in lambda calculus. The recursion implementation with anonymous function will be discussed later in this chapter.

In C#, recursion is a basic feature, so the following self reference is supported:

using static ChurchBoolean;

public static partial class ChurchNumeral
{
    // Divide = dividend => divisor => 
    //    If(dividend >= divisor)
    //        (_ => 1 + DivideBy(dividend - divisor)(divisor))
    //        (_ => 0);
    public static readonly Func<Numeral, Func<Numeral, Numeral>>
        DivideBy = dividend => divisor =>
            If(dividend.IsGreaterThanOrEqualTo(divisor))
                (_ => One.Add(DivideBy(dividend.Subtract(divisor))(divisor)))
                (_ => Zero);
}

Here using static directive is used so that ChurchBoolean.If function can be called directly. DivideBy is compiled to a field definition and field initialization code in static constructor, and apparently referencing to a field in the constructor is allowed:

using static ChurchBoolean;
using static ChurchNumeral;

public static partial class CompiledChurchNumeral
{
    public static readonly Func<Numeral, Func<Numeral, Numeral>> DivideBySelfReference;

    static CompiledChurchNumeral()
    {
        DivideBySelfReference = dividend => divisor =>
            If(dividend.IsGreaterThanOrEqualTo(divisor))
                (_ => One.Add(DivideBySelfReference(dividend.Subtract(divisor))(divisor)))
                (_ => Zero);
    }
}

The self reference also works for named function:

public static partial class ChurchNumeral
{
    public static Func<Numeral, Numeral> DivideByMethod(Numeral dividend) => divisor =>
        If(dividend.IsGreaterThanOrEqualTo(divisor))
            (_ => One.Add(DivideByMethod(dividend.Subtract(divisor))(divisor)))
            (_ => Zero);
}

The only exception is, when this function is a local variable instead of field, then the inline self reference cannot be compiled:

internal static void Inline()
{
    Func<Numeral, Func<Numeral, Numeral>> divideBy = dividend => divisor =>
        If(dividend.IsGreaterThanOrEqualTo(divisor))
            (_ => One.Add(divideBy(dividend.Subtract(divisor))(divisor)))
            (_ => Zero);
}

The reason is, the value of the local variable is compiled before the local variable is compiled. when the anonymous function is compiled, the referenced divideBy function is not defined yet, and C# compiler gives CS0165 error: Use of unassigned local variable 'divideBy'. To resolve this problem, divideBy can be first initialized with default value null. When divideBy is initialized again with the anonymous function, it is already defined, so the lambda expression can be compiled:

internal static void Inline()

{
    Func<Numeral, Func<Numeral, Numeral>> divideBy = null;
    divideBy = dividend => divisor =>
        If(dividend.IsGreaterThanOrEqualTo(divisor))
            (_ => One.Add(divideBy(dividend.Subtract(divisor))(divisor)))
            (_ => Zero);
}

The above division operator DivideBy will be used temporarily. Later after introducing fixed point combinator, the division can be implemented with an anonymous function without self reference at all.

Conversion between Church numeral and System.UInt32

In .NET, natural number can be represented with unit (System.UInt32). It would be intuitive if Church numeral and uint can be converted to each other. Similar to the conversion between Church Boolean and bool, the following extension methods can be defined:

public static partial class ChurchEncoding
{
    public static Numeral Church(this uint n) => n == 0U ? ChurchNumeral.Zero : Church(n - 1U).Increase();

    public static uint Unchurch(this Numeral n) => (uint)n(x => (uint)x + 1U)(0U);
}

Converting uint to Church numeral is recursive. When n is 0, Zero is returned directly. When n is positive, n is decreased and converted recursively. The recursion terminates when n is decreased to 0, then Increase is called for n times with Zero, and Church numeral n is calculated. And converting Church numeral n to uint just need to add 1U for n times to 0U.

The following code demonstrate how the operators and conversions work:

[TestClass]
public partial class ChurchNumeralTests
{
    [TestMethod]
    public void IncreaseTest()
    {
        Numeral numeral = 0U.Church();
        Assert.AreEqual(0U + 1U, (numeral = numeral.Increase()).Unchurch());
        Assert.AreEqual(1U + 1U, (numeral = numeral.Increase()).Unchurch());
        Assert.AreEqual(2U + 1U, (numeral = numeral.Increase()).Unchurch());
        Assert.AreEqual(3U + 1U, (numeral = numeral.Increase()).Unchurch());
        numeral = 123U.Church();
        Assert.AreEqual(123U + 1U, numeral.Increase().Unchurch());
    }

    [TestMethod]
    public void AddTest()
    {
        Assert.AreEqual(0U + 0U, 0U.Church().Add(0U.Church()).Unchurch());
        Assert.AreEqual(0U + 1U, 0U.Church().Add(1U.Church()).Unchurch());
        Assert.AreEqual(10U + 0U, 10U.Church().Add(0U.Church()).Unchurch());
        Assert.AreEqual(0U + 10U, 0U.Church().Add(10U.Church()).Unchurch());
        Assert.AreEqual(1U + 1U, 1U.Church().Add(1U.Church()).Unchurch());
        Assert.AreEqual(10U + 1U, 10U.Church().Add(1U.Church()).Unchurch());
        Assert.AreEqual(1U + 10U, 1U.Church().Add(10U.Church()).Unchurch());
        Assert.AreEqual(3U + 5U, 3U.Church().Add(5U.Church()).Unchurch());
        Assert.AreEqual(123U + 345U, 123U.Church().Add(345U.Church()).Unchurch());
    }

    [TestMethod]
    public void DecreaseTest()
    {
        Numeral numeral = 3U.Church();
        Assert.AreEqual(3U - 1U, (numeral = numeral.Decrease()).Unchurch());
        Assert.AreEqual(2U - 1U, (numeral = numeral.Decrease()).Unchurch());
        Assert.AreEqual(1U - 1U, (numeral = numeral.Decrease()).Unchurch());
        Assert.AreEqual(0U, (numeral = numeral.Decrease()).Unchurch());
        numeral = 123U.Church();
        Assert.AreEqual(123U - 1U, numeral.Decrease().Unchurch());
    }

    [TestMethod]
    public void SubtractTest()
    {
        Assert.AreEqual(0U - 0U, 0U.Church().Subtract(0U.Church()).Unchurch());
        Assert.AreEqual(0U, 0U.Church().Subtract(1U.Church()).Unchurch());
        Assert.AreEqual(10U - 0U, 10U.Church().Subtract(0U.Church()).Unchurch());
        Assert.AreEqual(0U, 0U.Church().Subtract(10U.Church()).Unchurch());
        Assert.AreEqual(1U - 1U, 1U.Church().Subtract(1U.Church()).Unchurch());
        Assert.AreEqual(10U - 1U, 10U.Church().Subtract(1U.Church()).Unchurch());
        Assert.AreEqual(0U, 1U.Church().Subtract(10U.Church()).Unchurch());
        Assert.AreEqual(0U, 3U.Church().Subtract(5U.Church()).Unchurch());
        Assert.AreEqual(0U, 123U.Church().Subtract(345U.Church()).Unchurch());
    }

    [TestMethod]
    public void MultiplyTest()
    {
        Assert.AreEqual(0U*0U, 0U.Church().Multiply(0U.Church()).Unchurch());
        Assert.AreEqual(0U*1U, 0U.Church().Multiply(1U.Church()).Unchurch());
        Assert.AreEqual(10U*0U, 10U.Church().Multiply(0U.Church()).Unchurch());
        Assert.AreEqual(0U*10U, 0U.Church().Multiply(10U.Church()).Unchurch());
        Assert.AreEqual(1U*1U, 1U.Church().Multiply(1U.Church()).Unchurch());
        Assert.AreEqual(10U*1U, 10U.Church().Multiply(1U.Church()).Unchurch());
        Assert.AreEqual(1U*10U, 1U.Church().Multiply(10U.Church()).Unchurch());
        Assert.AreEqual(3U*5U, 3U.Church().Multiply(5U.Church()).Unchurch());
        Assert.AreEqual(12U*23U, 12U.Church().Multiply(23U.Church()).Unchurch());
    }

    [TestMethod]
    public void PowTest()
    {
        Assert.AreEqual(Math.Pow(0U, 1U), 0U.Church().Pow(1U.Church()).Unchurch());
        Assert.AreEqual(Math.Pow(10U, 0U), 10U.Church().Pow(0U.Church()).Unchurch());
        Assert.AreEqual(Math.Pow(0U, 10U), 0U.Church().Pow(10U.Church()).Unchurch());
        Assert.AreEqual(Math.Pow(1U, 1U), 1U.Church().Pow(1U.Church()).Unchurch());
        Assert.AreEqual(Math.Pow(10U, 1U), 10U.Church().Pow(1U.Church()).Unchurch());
        Assert.AreEqual(Math.Pow(1U, 10U), 1U.Church().Pow(10U.Church()).Unchurch());
        Assert.AreEqual(Math.Pow(3U, 5U), 3U.Church().Pow(5U.Church()).Unchurch());
        Assert.AreEqual(Math.Pow(5U, 3U), 5U.Church().Pow(3U.Church()).Unchurch());
    }

    [TestMethod]
    public void DivideByRecursionTest()
    {
        Assert.AreEqual(1U / 1U, 1U.Church().DivideBy(1U.Church()).Unchurch());
        Assert.AreEqual(1U / 2U, 1U.Church().DivideBy(2U.Church()).Unchurch());
        Assert.AreEqual(2U / 2U, 2U.Church().DivideBy(2U.Church()).Unchurch());
        Assert.AreEqual(2U / 1U, 2U.Church().DivideBy(1U.Church()).Unchurch());
        Assert.AreEqual(10U / 3U, 10U.Church().DivideBy(3U.Church()).Unchurch());
        Assert.AreEqual(3U / 10U, 3U.Church().DivideBy(10U.Church()).Unchurch());
    }
}

Lambda Calculus via C# (2) Church Encoding: Boolean and Logic

Mon, 04 Nov 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Lambda Calculus via C# series]

Lambda calculus is a formal system for function definition and function application, so in lambda calculus, the only primitive is anonymous function. Anonymous function is actually very powerful. With an approach called Church encoding. data and operation can be modeled by higher-order anonymous functions and their application. Church encoding is named after Alonzo Church, who first discovered this approach. This part discusses Church Boolean - modeling Boolean values and logic operators with functions.

Church Boolean

Boolean values True and False can be both represented by anonymous function with 2 parameters. True function simply output the first parameter, and False function output the second parameter:

True := λtf.t
False := λtf.f

As fore mentioned, λtf.E is just the abbreviation of λt.λf.E, so these definitions actually are:

True := λt.λf.t
False := λt.λf.f

In this tutorial, for consistency and intuition, function definition with multiple variables is always represented in the latter curried form. In C#, they can be viewed as t => f => t and t => f => f, which are curried from (t, f) => t and (t, f) => f. Here t and f can be of any type, so leave their types as dynamic for convenience. In C#, at compile time dynamic is viewed as object and also supports any operation; at runtime if the operation is actually not supported, an exception is thrown. So, the function type of t => f => t and t => f => f is dynamic –> dynamic –> dynamic, which is represented as Func<dynamic, Func<dynamic, dynamic>> in C#. For convenience, an alias Boolean can be defined for such function type:

// Curried from (dynamic, dynamic) -> dynamic.
// Boolean is the alias of dynamic -> dynamic -> dynamic.
public delegate Func<dynamic, dynamic> Boolean(dynamic @true);

So that True and False can be defined with lambda expression:

public static partial class ChurchBoolean
{
    public static readonly Boolean
        True = @true => @false => @true;

    public static readonly Boolean
        False = @true => @false => @false;
}

C# does not support defining function directly in the global scope, so True and False are defined as static filed member of a type. In other functional languages like F#, functions can directly defined:

let True t f = t
let False t f = f

There is no noise and the function currying is default. Actually this F# code is compiled to CIL code similar to above C# structure (static member of a type).

Logical operators

After defining Boolean values True and False with functions, now the Boolean logics can be represented by functions too. And can be defined by the following function:

And := λa.λb.a b False

Applying function True with Boolean a and b:

When a is True, the application is beta reduced to True b False, which applies True function with b and False, and the first argument b is returned. In C#, this can be viewed that true && b is the same as b.
When a is False, the application is beta reduced to False b False, which applies False function with b and False, and the second argument False is returned. In C#, this can be viewed as false && b is always false.

And True b
≡ (λa.λb.a b False) True b
≡ (λb.True b False) b
≡ True b False
≡ b

  And False b
≡ (λa.λb.a b False) False b
≡ (λb.False b False) b
≡ False b False
≡ False

In C#, And can be viewed as a => b => a(b)(False), it is of curried function type Boolean –> Boolean -> Boolean:

public static partial class ChurchBoolean
{
    public static readonly Func<Boolean, Func<Boolean, Boolean>>
        And = a => b => a(b)(False);
}

This demonstrates that the Boolean alias improves the readability. Without this alias, the type of And becomes (dynamic –> dynamic –> dynamic) –> (dynamic –> dynamic –> dynamic) –> (dynamic –> dynamic –> dynamic), which is Func<Func<dynamic, Func<dynamic, dynamic>>, Func<Func<dynamic, Func<dynamic, dynamic>>, Func<dynamic, Func<dynamic, dynamic>>>> in C#.

This also demonstrates that dynamic type simplifies type conversion. If Boolean is defined as object –> object -> object:

public delegate Func<object, object> Boolean(object @true);

public static partial class ChurchBoolean
{
    public static readonly Func<Boolean, Func<Boolean, Boolean>>
        And = a => b => (Boolean)a(b)(False);
}

And must return Boolean, but a(b)(False) returns object, so a type conversion is required. Here a is either True or False, according to the definition of True and False, a(b)(False) returns either b or False. Since b and False are both of type Boolean, so here it is safe to convert a(b)(False) to Boolean. In contrast, when Boolean is defined as dynamic –> dynamic -> dynamic, a(b)(False) returns dynamic, which is viewed as supporting any operation at compile time, including implicitly conversion to Boolean, so the explicit type conversion is not required. At run time, a(b)(False) always return Boolean, and converting Boolean to Boolean always succeeds, so And works smoothly without any exception.

In the above lambda function and C# function, a function name False is referenced. Again, function is anonymous by default in lambda calculus. This tutorial uses function name only for for readability. By substituting function name, And can be defined as:

And := λa.λb.a b (λt.λf.f)

And the C# implementation becomes:

public static Func<Boolean, Func<Boolean, Boolean>>
    And = a => b => a(b)(new Boolean(@true => @false => @false));

The function body is longer and less readable. Also, a is of type dynamic –> dynamic -> dynamic, the second argument of a is expected to be object. When function reference False is given, False is a Boolean delegate instance, apparently it is an object and works there, However, when an inline C# lambda expression is given. C# compiler cannot infer the the type of this lambda expression – it could be anonymous function, or expression tree, and the type information of @true and @false cannot be inferred either. So here the constructor syntax is used to indicate this inline lambda expression is a function of type dynamic –> dynamic -> dynamic.

Again, C# does not support defining custom operators for functions, so a && operator cannot be defined for Boolean type. However, extension method can be defined for Boolean type, also And can be implemented as:

public static partial class BooleanExtensions
{
    public static Boolean And(this Boolean a, Boolean b) => ChurchBoolean.And(a)(b);
}

Now And can be used fluently like an infix operator:

internal static void CallAnd()
{
    Boolean result1 = True.And(True);

    Boolean x = True;
    Boolean y = False;
    Boolean result2 = x.And(y);
}

Once again, the function name And is only for readability, without refereeing to the function name., the function application (And x y) has to be written as (λa.λb.a b (λt.λf.f)) x y, and in C#, calling And anonymously works but is also less readable:

internal static void CallAnonymousAnd()
{
    Boolean result1 = new Func<Boolean, Func<Boolean, Boolean>>(a => b => (Boolean)a(b)(False))(True)(True);

    Boolean x = True;
    Boolean y = False;
    Boolean result2 = new Func<Boolean, Func<Boolean, Boolean>>(a => b => (Boolean)a(b)(False))(x)(y);
}

Or is defined as:

Or :=  λa.λb.a True b

When a is True, True True b returns the first argument True; When a is False, False True b returns the second argument b. In C#, this can be viewed as true || b is always true, and false || b is the same as b.

Or True b
≡ (λa.λb.a True b) True b
≡ (λb.True True b) b
≡ True True b
≡ True
 
  Or False b
≡ (λa.λb.a True b) False b
≡ (λb.False True b) b
≡ False True b
≡ b

Not is defined as:

Not := λa.a False True

When a is True, True False True returns the first argument False; when a is False, False False True returns the second argument True:

Not True
≡ (λa.a False True) True
≡ True False True
≡ False
 
  Not False
≡ (λa.a False True) False
≡ False False True
≡ True

Xor is defined as:

Xor := λa.λb.a (Not b) b

When a is True, True (Not b) b returns the first argument Not b; when a is False, True (Not b) b returns the second argument b:

Xor True b
≡ (λa.λb.a (Not b) b) True b
≡ (λb.True (Not b) b) b
≡ True (Not b) b
≡ Not b
 
  Xor False b
≡ (λa.λb.a (Not b) b) True b
≡ (λb.False (Not b) b) b
≡ False (Not b) b
≡ b

These 3 operators can be simply implemented as:

public static Func<Boolean, Func<Boolean, Boolean>> 
    Or = a => b => a(True)(b);

public static Func<Boolean, Boolean> 
    Not = boolean => boolean(False)(True);

public static Func<Boolean, Func<Boolean, Boolean>>
    Xor = a => b => a(Not(b))(b);

Again, they can be wrapped as extension methods too:

public static Boolean Or(this Boolean a, Boolean b) => ChurchBoolean.Or(a)(b);

public static Boolean Not(this Boolean a) => ChurchBoolean.Not(a);

public static Boolean Xor(this Boolean a, Boolean b) => ChurchBoolean.Xor(a)(b);

Conversion between Church Boolean and System.Boolean

It could be intuitive if the Church Boolean function can be directly compared with .NET bool value. The following methods can be defined to convert between them:

public static partial class ChurchEncoding
{
    // System.Boolean structure to Boolean function.
    public static Boolean Church(this bool boolean) => boolean ? True : False;

    // Boolean function to System.Boolean structure.
    public static bool Unchurch(this Boolean boolean) => boolean(true)(false);
}

With the help of conversion, the following code demonstrate how to use the logical operators:

[TestClass]
public partial class ChurchBooleanTests
{
    [TestMethod]
    public void NotTest()
    {
        Assert.AreEqual((!true).Church(), True.Not());
        Assert.AreEqual((!false).Church(), False.Not());
    }

    [TestMethod]
    public void AndTest()
    {
        Assert.AreEqual((true && true).Church(), True.And(True));
        Assert.AreEqual((true && false).Church(), True.And(False));
        Assert.AreEqual((false && true).Church(), False.And(True));
        Assert.AreEqual((false && false).Church(), False.And(False));
    }

    [TestMethod]
    public void OrTest()
    {
        Assert.AreEqual((true || true).Church(), True.Or(True));
        Assert.AreEqual((true || false).Church(), True.Or(False));
        Assert.AreEqual((false || true).Church(), False.Or(True));
        Assert.AreEqual((false || false).Church(), False.Or(False));
    }

    [TestMethod]
    public void XorTest()
    {
        Assert.AreEqual((true ^ true).Church(), True.Xor(True));
        Assert.AreEqual((true ^ false).Church(), True.Xor(False));
        Assert.AreEqual((false ^ true).Church(), False.Xor(True));
        Assert.AreEqual((false ^ false).Church(), False.Xor(False));
    }
}

If

The if logic is already built in Church Booleans. Church Booleans is a function that can be applied with 2 argument. If this Church Boolean function is True, the first argument is returned, else the second argument is returned. So naturedly, the following is the If function, which is just a wrapper of Church Boolean function application:

If := λb.λt.λf.b t f

The first argument b is a Church Boolean. when b is True, If returns second argument t. When b is False, If returns third argument f. In C#:

// EagerIf = condition => then => @else => condition(then)(@else)
public static readonly Func<Boolean, Func<dynamic, Func<dynamic, dynamic>>>
    EagerIf = condition => then => @else =>
        condition    // if (condition)
            (then)   // then { ... }
            (@else); // else { ... }

There is one issue with this C# implementation. As fore mentioned, C#’s reduction strategy is applicative order, when C# function is called, arguments are evaluated, then function is called:

internal static void CallEagerIf(Boolean condition, Boolean a, Boolean b)
{
    Boolean result = EagerIf(condition)
        (a.And(b)) // then branch.
        (a.Or(b)); // else branch.
}

In this example, disregarding condition is True or False, the then branch a.And(b) and else branch a.Or(b) are both executed. If would be better if one branch is executed for a certain condition. The solution is to make If’s second and third arguments of type T to a factory of type Unit<T> –> T:

// If = condition => thenFactory => elseFactory => condition(thenFactory, elseFactory)(Id)
public static readonly Func<Boolean, Func<Func<Unit<dynamic>, dynamic>, Func<Func<Unit<dynamic>, dynamic>, dynamic>>>
    If = condition => thenFactory => elseFactory =>
        condition
            (thenFactory)
            (elseFactory)(Functions<dynamic>.Id);

In lambda calculus this is equivalent to:

If := λb.λt.λf.b t f Id

Now calling If becomes:

internal static void CallLazyIf(Boolean condition, Boolean a, Boolean b)
{
    Boolean result = If(condition)
        (id => a.And(b)) // then.
        (id => a.Or(b)); // else.
}

When condition is True, only a.And(b) is executed. When condition is False, only a.Or(b) is executed. Now the then and else branches are represented by factory functions id => a.And(b) and id => a.Or(b), where the id argument is the Id function. This argument usually is not used by the function body, it can be named as _ to indicate “don’t care”:

internal static void CallLazyIf(Boolean condition, Boolean a, Boolean b)
{
    Boolean result = If(condition)
        (_ => a.And(b)) // then.
        (_ => a.Or(b)); // else.
}

Lambda Calculus via C# (1) Fundamentals

Fri, 01 Nov 2024 00:00:00 GMT

[FP & LINQ via C# series]

[Lambda Calculus via C# series]

Lambda calculus (aka λ-calculus) is a theoretical framework to describe function definition, function application, function recursion, and uses functions and function application to express computation. It is a mathematics formal system, but can also be viewed as a smallest programming language that can express and evaluate any computable function. As an universal model of computation, lambda calculus is important in programming language theory, and especially it is the foundation of functional programming. The knowledge of lambda calculus greatly helps understanding functional programming, LINQ, C# and other functional languages.

Expression

The core concept of lambda calculus is expression. There are 3 kinds of expressions in lambda calculus: variable, function, application. Expression can be defined recursively:

If v is a variable, then v is expression
If v is a variable and E is expression, then function λv.E is expression. The function syntax λv.E can be viewed as the the C# anonymous function syntax v => E, where v is the parameter and E is the function body expression.
If E1 is expression and E2 is expression, then E1 E2 is expression, which is called application. The application syntax E1 E2 can be viewed as C# function call syntax E1(E2), where E1 is the function definition expression and E2 is the argument expression.

By default, lambda calculus treat function anonymously. There is only variable name in lambda calculus. There is no function name involved in function definition expression. In C# language, lambda expression representing anonymous function is a feature introduced in C# 3.0 with .NET Framework 3.5 years back. Actually the theory of lambda expression and lambda calculus were introduced as early as 1930s by Alonzo Church, a mathematician and the doctoral advisor of Alan Turing.

The following are conventions of expression:

Outermost parentheses can be dropped, e.g. E1 E2 means (E1 E2), in C# it can be viewed as (E1(E2)): call function E1 with argument E2
A sequence of functions is contracted: , e.g. sequence of function λx.(λy.(λz.E)) is contracted as λxyz.E, in another word, expression λxyz.E actually means λx.(λy.(λz.E)), which is identical to λx.λy.λz.E because the parentheses are not required. In C# it can be viewed that (x, y, z) => E is always curried to x => (y => (z => E)), which is identical to x => y => z => E because => operator is right associative
Application is left associative, e.g. E1 E2 E3 means ((E1 E2) E3), in C# it can be viewed as ((E1(E2))(E3)): call function E1 with argument E2, then call the returned function with argument E3

Bound variable vs. free variable

In function, its body expression can use variables. There are 2 kinds of variables used in function body expression, bound variable and free variable:

When function’s variable (variables before . symbol) occurs in the function body expression, these these variable occurrences (after the . symbol) are bound variables. In C# this can be viewed as declared function parameter’s occurrences in function body.
All other variables are free variables, in C# it can be viewed as outer variable or closure.

For example, for function λx.f x, its body expression f x has bound variable x, and free variable f. This can be viewed as x => f(x) in C# syntax, in the body x is parameter and f is closure.

A variable is bound by its "nearest" function. For example, in λx.g x (λx.h x), the first occurrence of x in the body expression is bound by the outer function, and the second occurrence of x is bound by the inner function. In C#, x => g(x)(x => h(x)) cannot be compiled for this reason - the outer function parameter has the same name as the inner function parameter, which is disallowed by C# compiler:

internal static class Expression
{
    internal static Func<T, T> Variable<T>(Func<T, Func<Func<T, T>, T>> g, Func<T, T> h) => 
        x => g(x)(x => h(x));
}

Expression without free variables are also called combinator, which will be discussed later.

Reduction

In lambda calculus, there are 3 substitution rules for expression to be reduced.

α-conversion

In lambda calculus, lambda expression’s bound variables can be substituted with different name. This is called alpha-conversion, or alpha-renaming. In C#, this can be viewed as function parameter can be renamed, for example, x => f(x) is equivalent to y => f(y).

In the above example of λx.g x (λx.h x), the inner function λx.h x has variable x, which can be substituted with a different name y, along with its appearance in the body h x. Then the inner function becomes λy.h y, so the outer function becomes λx.g x (λy.h y). Now it becomes intuitive how x and y are bound by the “nearest” function. In C#, x => g(x)(y => h(y)) can be compiled:

internal static Func<T, T> Variable<T>(Func<T, Func<Func<T, T>, T>> g, Func<T, T> h) => 
    x => g(x)(y => h(y));

β-reduction

Beta-reduction of function application expression (λv.E) R is denoted E[v := R]. It means to substitute all free occurrences of the variable v in the expression E with expression R. In C#, this can be viewed as when function is called with argument, in the body all parameter occurrences are substituted by argument. For Example, when function x => x + 2 is called with 1, in the body x + 2, parameter x is substituted with argument 1, so the function is evaluated to 1 + 2.

η-conversion

Eta-conversion means 2 functions are the same if and only if they always give the same result for the same argument. For example λx.f x can be substituted with f, if x does not appear free in f. In C#, this can be viewed as that function x => f(x) is equivalent to function f. For example:

internal static void LinqQuery()
{
    Func<int, bool> isEven = value => value % 2 == 0;
    Enumerable.Range(0, 5).Where(value => isEven(value)).ForEach(value => Console.WriteLine(value));
}

Here function value => isEven(value) and function isEven always have the same result for the same argument, so value=> isEven(value) can be substituted with isEven. Similarly value => Console.WriteLine(value) can be substituted by Console.WriteLine. The above LINQ query is equivalent to:

internal static void EtaConvertion()
{
    Func<int, bool> isEven = value => value % 2 == 0;
    Enumerable.Range(0, 5).Where(isEven).ForEach(Console.WriteLine);
}

Normal order

The above reduction rules can be applied to expression with different order. With normal order, the leftmost, outermost expression is reduced first. For function application expression, this means the function is beta reduced first, then the arguments are reduced, for example:

(λx.λy.y) ((λa.λb.a) (λv.v))
≡ λy.λy

In this expression, function (λx.λy.y) is applied with argument, expression ((λa.λb.a) (λv.v)). The leftmost, outermost expression is the function expression (λx.λy.y). So in its body λy.y, all free occurrences of x should be substituted by ((λa.λb.a) (λv.v)). And since there is no any occurrences of x, the substitution result is still λy.y. In normal order reduction, the argument expression ((λa.λb.a) (λv.v)) is not reduced at all.

Here λy.y cannot be further reduced. An expression that cannot be reduced any further with above 3 rules is called in normal form. Here λy.λy is the normal form of (λx.λy.y) ((λa.λb.a) (λv.v)). Some lambda expressions can be reduced infinitely so does not have normal form, which will be discussed later.

Applicative order

With applicative order, the rightmost, innermost expression is reduced first. For function application expression, this means the arguments are reduced first, then the function is beta reduced. Take the above expression as example again:

(λx.λy.y) ((λa.λb.a) (λv.v))
≡ (λx.λy.y) (λb.λv.v)
≡ λy.λy

The argument expression ((λa.λb.a) (λv.v)) is righter than the function definition expression (λx.λy.y), so ((λa.λb.a) (λv.v)) is reduced first. It can be beta reduced to normal form (λb.λv.v), which cannot be further reduced. Then (λx.λy.y) is applied with (λb.λv.v), which can be beta reduced to normal form λy.λy. In application order reduction, argument must be reduced before function application. This is the strategy of C#.

In lambda calculus, reducing expression in any order produces the same result, which is the Church–Rosser theorem.

Function composition

In lambda calculus function composition means to combine simple functions into a more complicated function, which can be viewed the same as fore mentioned C# function composition. The composition of f1 and f2 is denoted f2 ∘ f1. This new function (f2 ∘ f1)’s application is defined as:

(f2 ∘ f1) x := f2 (f1 x)

Here the function names f1 and f2 indicate the order of being applied. f2 ∘ f1 can also be read as f2 after f1. in C#, this can be viewed as the forward composition discussed before:

public static partial class FuncExtensions
{
    public static Func<T, TResult2> After<T, TResult1, TResult2>(
        this Func<TResult1, TResult2> function2, Func<T, TResult1> function1) =>
            value => function2(function1(value));
}

As fore mentioned, some other functional languages have built in composition operator for functions, like >> in F#, . in Haskell, etc. C# does not support defining custom operators for functions. As a workaround, an extension method o can be defined to represent this ∘ operator:

public static Func<T, TResult2> o<T, TResult1, TResult2>(
    this Func<TResult1, TResult2> function2, Func<T, TResult1> function1) =>
        value => function2(function1(value));

So that f3 ∘ f2 ∘ f1 becomes f3.o(f2).o(f1) in C#, which is more intuitive, for example:

internal static void Compose()
{
    Func<double, double> sqrt = Math.Sqrt;
    Func<double, double> abs = Math.Abs;

    Func<double, double> absSqrt1 = sqrt.o(abs); // Composition: sqrt after abs.
    absSqrt1(-2D).WriteLine(); // 1.4142135623731
}

Associativity

Function composition is associative. That means (f3 ∘ f2) ∘ f1 and f3 ∘ (f2 ∘ f1) are equivalent.

When applying x to (f3 ∘ f2) ∘ f1, according to the definition of ∘:

((f3 ∘ f2) ∘ f1) x
≡ (f3 ∘ f2) (f1 x)
≡ f3 (f2 (f1 x))

And when applying x to f3 ∘ (f2 ∘ f1):

f3 ∘ (f2 ∘ f1) x
≡ f3 ∘ (f2 (f1 x))
≡ f3 (f2 (f1 x))

In C#, this means f3.o(f2).o(f1) and f3.o(f2.o(f1)) are equivalent:’

internal static void Associativity()
{
    Func<double, double> sqrt = Math.Sqrt;
    Func<double, double> abs = Math.Abs;
    Func<double, double> log = Math.Log;

    Func<double, double> absSqrtLog1 = log.o(sqrt).o(abs); // Composition: (log o sqrt) o abs.
    absSqrtLog1(-2D).WriteLine(); // 0.34642256747438094
    Func<double, double> absSqrtLog2 = log.o(sqrt.o(abs)); // Composition: log o (sqrt o abs).
    absSqrtLog2(-2D).WriteLine(); // 0.34642256747438094
}

Unit

There is a unit function Id for function composition:

Id := λx.x

so that f ∘ Id and Id ∘ f are both equivalent to f:

f ∘ Id = f
Id ∘ f = f

According to the definition of ∘ and Id:

(f ∘ Id) x
≡ f (Id x)
≡ f x

  (Id ∘ f) x
≡ Id (f x)
≡ f x

In C#, Id can be defined as:

// Unit<T> is the alias of Func<T, T>.
public delegate T Unit<T>(T value);

public static partial class Functions<T>
{
    public static readonly Unit<T>
        Id = x => x;
}

Here function expression (λx.x) is given a name Id, this is only for readability. Later, when refereeing to this function, its name Id will used, which is more intuitive than the lambda expression.

Functional Programming and LINQ via C#

Wed, 14 Aug 2024 00:00:00 GMT

Acclaim
Contents at a Glance
Table of Contents

This is a book on functional programming and LINQ programming via C# language. It discusses:

Functional programming via C# in-depth
Use functional LINQ to work with local data and cloud data
The underlying mathematics theories of functional programming and LINQ, including Lambda Calculus and Category Theory

Acclaim

Microsoft:

“An excellent book for those of us who need to get in-depth understanding on LINQ and functional programming with latest C# language. The author made sure this book includes the latest and cross-platform knowledge for the language, the framework, as well as the underlying mathematical theories.” Hongfei Guo Partner Group Engineering Manager at Microsoft
“This book explains practical and in-depth material clearly, concisely and accurately to the areas of the C# language, functional programming, and LINQ on .NET Framework and .NET Core. This is a great book for anyone wanting to understand the whys and hows behind these important technologies.” Samer Boshra Principal Software Development Engineer at Microsoft
“This is a great book for developers who want to go functional programming. It's one-stop shopping for serious developers who have to get up to speed with LINQ and functional programming quickly and in-depth. I'll keep this book on my desk not on my bookshelf.” Roshan Kommusetty Principal Software Engineering Manager at Microsoft
“This is a great book for C# developers, it covers both basic C# programming concepts for the beginners new to the .NET world, and C# advanced constructs for experienced .NET programmers. The book is up to date, talks C# 7.0 new language features and demonstrates how you can use them for functional programming. Thanks for the awesome work!” Mark Zhou Principal Software Engineering Manager at Microsoft
“I like the way the author presented the detailed knowledge with a lot of examples. As a data scientist with statistics background in a number of industries, I can pick up C# programming and LINQ quickly when I followed the book. The book was concise and easy to read. It was a pleasant experience for me to spend my time emerging myself in the book in the sunshine weekday afternoon.” Xue Liu Senior Data Scientist at Microsoft
“Functional Programming and LINQ in C# language, have been fully and clearly unraveled in this book, with many practical examples. The author has not saved any effort to go beyond scratching the surface of C# language and has successfully explained the magic behind the scene. This book is a must-have for anyone who wants to understand functional programming using C#.” Jie Mei Data & Applied Scientist at Microsoft

Academy and more:

“This book provides comprehensive and in-depth information about the C# functional programming and LINQ technologies to application developers on both .NET Framework and .NET Core. The detailed text and wealth of examples will give a developer a clear and solid understanding of C# language, functional programming and using LINQ to work with different data domains.” Dong Si Assistant Professor, Department of Computer Science, University of Washington, Bothell
“This book offers a comprehensive, in-depth, yet easy-to-understand tutorial to functional C# programming and LINQ. Filled with detailed explanations and real-world examples, this book is highly valuable for beginners and experienced developers alike.” Shuang Zhao Assistant Professor, Department of Computer Science, University of California, Irvine
“This excellent book is an in-depth and also readable exploration of C# functional programming and LINQ programming. It covers .NET Framework and .NET Core in great detail.” Yang Sha Engineering Manager at Google
“Great book! It takes a hands-on approach to LINQ and functional programming in an easy to understand format. I would highly recommend this book to developers looking to develop expertise in C#, functional programming, and LINQ.” Himanshu Lal Software Engineering Manager at Facebook
“This is a great book that combines practical examples with in-depth analysis of LINQ and functional programming in C#. Dixin leverages his expertise in .NET to provide a well written tutorial on the effective use of LINQ and an overview of the theoretical principles behind it. A must read for anyone working on these technologies!” Dimitrios Soulios Director at Goldman Sachs

Contents at a Glance

The contents are organized as the following chapters:

Part 1 Code - covers functional programming via C#, and fundamentals of LINQ.
- Chapter 1 Functional programming and LINQ paradigm
  - What is LINQ, how LINQ uses language to work with many different data domains.
  - Programming paradigm, imperative vs. declarative programming, object-oriented vs. functional programming.
- Chapter 2 Functional programming in depth
  - C# fundamentals for beginners.
  - Aspects of functional programming via C#, including function type, named/anonymous/local function, closure, lambda, higher-order function, currying, partial application, first class function, function composition, query expression, covariance/contravariance, immutability, tuple, purity, async function, pattern matching, etc., including how C# is processed at compile time and runtime.
Part 2 Data - covers how to use functional LINQ to work with different data domains in the real world, and how LINQ works internally.
- Chapter 3 LINQ to Objects
  - How to use functional LINQ queries to work with objects, covering all LINQ and Ix.
  - How the LINQ to Objects query methods are implemented, how to implement useful custom LINQ queries.
- Chapter 4 LINQ to XML
  - How to modeling XML data, and use functional LINQ queries to work with XML data.
  - How to use the other LINQ to XML APIs to manipulate XML data.
- Chapter 5 Parallel LINQ
  - How to use parallelized functional LINQ queries to work with objects.
  - Performance analysis for parallel/sequential LINQ queries.
- Chapter 6 Entity Framework/Core and LINQ to Entities
  - How to model database with object-relational mapping, and use functional LINQ queries to work with relational data in database.
  - How the C# LINQ to Entities queries are implemented to work with database.
  - How to change data in database, and handle concurrent conflicts.
  - Performance tips and asynchrony.
Part 3 Theories - demystifies the abstract mathematics theories, which are the rationale and foundations of LINQ and functional programming.
- Chapter 7 Lambda Calculus via C#
  - Core concepts of lambda calculus, bound and free variables, reduction (α-conversion, β-reduction, η-conversion), etc.
  - How to use lambda functions to represent values, data structures and computation, including Church Boolean, Church numbers, Church pair, Church list, and their operations.
  - Combinators and combinatory logic, including SKI combinator calculus, fixed point combinator for function recursion, etc.
- Chapter 8 Category Theory via C#
  - Core concepts of category theory, including category, object, morphism, monoid, functor, natural transformation, applicative functor, monad, and their laws.
  - How these concepts are applied in functional programming and LINQ.
  - How to manage I/O, state, exception handling, shared environment, logging, and continuation, etc., in functional programming.

This tutorial delivers highly reusable knowledge:

It covers C# language in depth, which can be generally applied in any programming paradigms besides functional programming.
It is a cross platform tutorial, covering both .NET Framework for Windows and .NET Core for Windows, Mac, Linux.
It demonstrates both usage and implementation of LINQ for mainstream data domains, which also enables developer to use the LINQ technologies for other data domains, or build custom LINQ APIs for specific data scenarios.
It also demystifies the abstract mathematics knowledge for functional programming, which applies to general functional programming, so it greatly helps developers understanding any other functional languages too.

As a fun of functional programming, LINQ, C#, and .NET technologies, hope this helps.

All code examples are available on GitHub: https://github.com/Dixin/CodeSnippets.

Functional programming and LINQ paradigm
1. Cross platform C# and .NET
  - Introducing cross platform .NET, C# and LINQ
    - .NET Framework, C#, and LINQ
    - .NET Core, UWP, Mono, Xamarin and Unity
    - .NET Standard
  - Introducing this book
    - Book structure
    - Code examples
  - Start coding
    - Start coding with Visual Studio (Windows)
    - Start coding with Visual Studio Code (Windows, macOS and Linux)
    - Start coding with Visual Studio for Mac (macOS)
2. Programming paradigms and functional programming
  - Programming paradigms
  - Imperative programming vs. declarative programming
  - Object-oriented programming vs. functional programming
3. LINQ to data sources
  - One language for different data domains
    - LINQ to Objects
    - Parallel LINQ
    - LINQ to XML
    - LINQ to DataSets
    - LINQ to Entities
    - LINQ to SQL
    - LINQ to NoSQL (LINQ to CosmosDB)
    - LINQ to JSON
    - LINQ to Twitter
  - Sequential query vs. parallel query
  - Local query vs. remote query
Functional programming in depth
1. C# language basics
 - Types and members
 - Types and members
 - Built-in types
 - Reference type vs. value type
 - ref local variable and immutable ref local variable
 - Array and stack-allocated array
 - Default value
 - ref structure
 - Static class
 - Partial type
 - Interface and implementation
 - IDisposable interface and using declaration
 - Generic type
 - Type parameter
 - Type parameter constraints
 - Nullable value type
 - Auto property
 - Property initializer
 - Object initializer
 - Collection initializer
 - Index initializer
 - Null coalescing operator
 - Null conditional operator
 - throw expression
 - Exception filter
 - String interpolation
 - nameof operator
 - Digit separator and leading underscore
2. Named function and function polymorphism
 - Constructor, static constructor and finalizer
 - Static method and instance method
 - Extension method
 - More named functions
 - Function polymorphisms
 - Ad hoc polymorphism: method overload
 - Parametric polymorphism: generic method
 - Type argument inference
 - Static import
 - Partial method
3. Local function and closure
 - Local function
 - Closure
 - Outer variable
 - Implicit reference
 - Static local function
4. Function input and output
 - Input by copy vs. input by alias (ref parameter)
 - Input by immutable alias (in parameter)
 - Output parameter (out parameter) and out variable
 - Discarding out variable
 - Parameter array
 - Positional argument vs. named argument
 - Required parameter vs. optional parameter
 - Caller information parameter
 - Output by copy vs. output by alias
 - Output by immutable alias
5. Delegate: Function type, instance, and group
 - Delegate type as function type
 - Function type
 - Generic delegate type
 - Unified built-in delegate types
 - Delegate instance as function instance
 - Delegate instance as function group
 - Event and event handler
6. Anonymous function and lambda expression
 - Anonymous method
 - Lambda expression as anonymous function
 - IIFE (Immediately-invoked function expression)
 - Closure
 - Expression bodied function member
7. Expression tree: Function as data
 - Lambda expression as expression tree
 - Metaprogramming: function as abstract syntax tree
 - .NET expressions
 - Compile expression tree at runtime
 - Traverse expression tree
 - Expression tree to CIL at runtime
 - Expression tree to executable function at runtime
 - Expression tree and LINQ remote query
8. Higher-order function, currying and first class function
 - First order function vs. higher-order function
 - Convert first-order function to higher-order function
 - Lambda operator => associativity
 - Curry function
 - Uncurry function
 - Partial applying function
 - First-class function
9. Function composition and chaining
 - Forward composition vs. backward composition
 - Forward piping
 - Method chaining and fluent interface
10. LINQ query expression
 - Syntax and compilation
 - Query expression pattern
 - LINQ query expression
 - Forward piping with LINQ
 - Query expression vs. query method
11. Covariance and contravariance
 - Subtyping and type polymorphism
 - Variances of non-generic function type
 - Variances of generic function type
 - Variances of generic interface
 - Variances of generic higher-order function type
 - Covariance of array
 - Variances in .NET and LINQ
12. Immutability, anonymous type and tuple
 - Immutable value
 - Constant local
 - Enumeration
 - using declaration and foreach statement
 - Immutable alias (immutable ref local variable)
 - Function’s immutable input and immutable output
 - Range variable in LINQ query expression
 - this reference for class
 - Immutable state (immutable type)
 - Constant field
 - Immutable class with readonly instance field
 - Immutable structure (readonly structure)
 - Immutable anonymous type
 - Local variable type inference
 - Immutable tuple vs. mutable tuple
 - Construction, element name and element inference
 - Deconstruction
 - Tuple assignment
 - Immutable collection vs. readonly collection
 - Shallow immutability vs. deep immutability
13. Pure function
 - Pure function vs. impure function
 - Referential transparency and side effect free
 - Purity in .NET
 - Purity in LINQ
14. Asynchronous function
 - Task, Task<TResult> and asynchrony
 - Named async function
 - Awaitable-awaiter pattern
 - Async state machine
 - Runtime context capture
 - Generalized async return type and async method builder
 - ValueTask<TResult> and performance
 - Anonymous async function
 - Asynchronous sequence: IAsyncEnumerable<T>
 - async using declaration: IAsyncDispose
15. Pattern matching
 - Is expression
 - switch statement and switch expression
LINQ to Objects: Querying objects in memory
1. Local sequential LINQ query
 - Iteration pattern and foreach statement
 - IEnumerable<T> and IEnumerator<T>
 - foreach loop vs. for loop
 - Non-generic sequence vs. generic sequence
 - LINQ to Objects queryable types
2. LINQ to Objects standard queries and query expressions
 - Sequence queries
 - Generation: Empty , Range, Repeat, DefaultIfEmpty
 - Filtering (restriction): Where, OfType, where
 - Mapping (projection): Select, SelectMany, from, let, select
 - Grouping: GroupBy, group, by, into
 - Join
 - Inner join: Join, SelectMany, join, on, equals
 - Outer join: GroupJoin, join, into, on, equals
 - Cross join: SelectMany, Join, from select, join, on, equals
 - Concatenation: Concat
 - Set: Distinct, Union, Intersect, Except
 - Convolution: Zip
 - Partitioning: Take, Skip, TakeWhile, SkipWhile
 - Ordering: OrderBy, ThenBy, OrderByDescending, ThenByDescending, Reverse, orderby, ascending, descending, into
 - Conversion: Cast, AsEnumerable
 - Collection queries
 - Conversion: ToArray, ToList, ToDictionary, ToLookup
 - Value queries
 - Element: First, FirstOrDefault, Last, LastOrDefault, ElementAt, ElementAtOrDefault, Single, SingleOrDefault
 - Aggregation: Aggregate, Count, LongCount, Min, Max, Sum, Average
 - Quantifier: All, Any, Contains
 - Equality: SequenceEqual
 - Queries in other languages
3. Generator
 - Implementing iterator pattern
 - Generating sequence and iterator
 - Yield statement and generator
4. Deferred execution, lazy evaluation and eager Evaluation
 - Immediate execution vs. Deferred execution
 - Cold IEnumerable<T> vs. hot IEnumerable<T>
 - Lazy evaluation vs. eager evaluation
5. LINQ to Objects internals: Standard queries implementation
 - Argument check and deferred execution
 - Collection queries
 - Conversion: ToArray, ToList, ToDictionary, ToLookup
 - Sequence queries
 - Conversion: Cast, AsEnumerable
 - Generation: Empty , Range, Repeat, DefaultIfEmpty
 - Filtering (restriction): Where, OfType
 - Mapping (projection): Select, SelectMany
 - Grouping: GroupBy
 - Join: SelectMany, Join, GroupJoin
 - Concatenation: Concat
 - Set: Distinct, Union, Intersect, Except
 - Convolution: Zip
 - Partitioning: Take, Skip, TakeWhile, SkipWhile
 - Ordering: OrderBy, ThenBy, OrderByDescending, ThenByDescending, Reverse
 - Value queries
 - Element: First, FirstOrDefault, Last, LastOrDefault, ElementAt, ElementAtOrDefault, Single, SingleOrDefault
 - Aggregation: Aggregate, Count, LongCount, Min, Max, Sum, Average
 - Quantifier: All, Any, Contains
 - Equality: SequenceEqual
6. Advanced queries in Microsoft Interactive Extensions (Ix)
 - Sequence queries
 - Generation: Defer, Create, Return, Repeat
 - Filtering: IgnoreElements, DistinctUntilChanged
 - Mapping: SelectMany, Scan, Expand
 - Concatenation: Concat, StartWith
 - Set: Distinct
 - Partitioning: TakeLast, SkipLast
 - Conversion: Hide
 - Buffering: Buffer, Share, Publish, Memoize
 - Exception: Throw, Catch, Finally, OnErrorResumeNext, Retry
 - Imperative: If, Case, Using, While, DoWhile, Generate, For
 - Iteration: Do
 - Value queries
 - Aggregation: Min, Max, MinBy, MaxBy
 - Quantifiers: isEmpty
 - Void queries
 - Iteration: ForEach
7. Building custom queries
 - Sequence queries (deferred execution)
 - Generation: Create, RandomInt32, RandomDouble, FromValue, FromValues, EmptyIfNull
 - Filtering: Timeout
 - Concatenation: Join, Append, Prepend, AppendTo, PrependTo
 - Partitioning: Subsequence
 - Exception: Catch, Retry
 - Comparison: OrderBy, OrderByDescending, ThenBy, ThenByDescending, GroupBy, Join, GroupJoin, Distinct, Union, Intersect, Except
 - List: Insert, Remove, RemoveAll, RemoveAt
 - Collection queries
 - Comparison: ToDictionary, ToLookup
 - Value queries
 - List: IndexOf, LastIndexOf
 - Aggregation: PercentileExclusive, PercentileInclusive, Percentile
 - Quantifiers: IsNullOrEmpty, IsNotNullOrEmpty
 - Comparison: Contains, SequenceEqual
 - Void queries
 - Iteration: ForEach
LINQ to XML: Querying XML
1. Modeling XML
  - Imperative vs. declarative paradigm
  - Types, conversions and operators
  - Read and deserialize XML
  - Serialize and write XML
  - Deferred construction
2. LINQ to XML standard queries
  - Navigation
  - Ordering
  - Comparison
  - More useful queries
  - XPath
    - Generate XPath expression
3. Manipulating XML
  - Clone
  - Adding, deleting, replacing, updating, and events
  - Annotation
  - Validating XML with XSD
  - Transforming XML with XSL
Parallel LINQ: Querying objects in parallel
1. Parallel LINQ query and visualization
  - Parallel query vs. sequential query
  - Parallel query execution
  - Visualizing parallel query execution
    - Using Concurrency Visualizer
    - Visualizing sequential and parallel LINQ queries
    - Visualizing chaining query methods
2. Parallel LINQ internals: data partitioning
  - Partitioning and load balancing
    - Range partitioning
    - Chunk partitioning
    - Hash partitioning
    - Stripped partitioning
  - Implement custom partitioner
    - Static partitioner
    - Dynamic partitioner
3. Parallel LINQ standard queries
  - Query settings
    - Cancellation
    - Degree of parallelism
    - Execution mode
    - Merge the values
  - Ordering
    - Preserving the order
    - Order and correctness
    - Orderable partitioner
  - Aggregation
    - Commutativity, associativity and correctness
    - Partitioning and merging
4. Parallel query performance
  - Sequential query vs. parallel query
  - CPU bound operation vs. IO bound operation
  - Factors to impact performance
Entity Framework/Core and LINQ to Entities: Querying relational data
1. Remote LINQ query
 - Entity Framework and Entity Framework Core
 - SQL database
 - Remote query vs. local query
 - Function vs. expression tree
2. Modeling database with object-relational mapping
 - Data types
 - Database
 - Connection resiliency and execution retry strategy
 - Tables
 - Relationships
 - One-to-one
 - One-to-many
 - Many-to-many
 - Inheritance
 - Views
3. Logging and tracing LINQ to Entities queries
 - Application side logging
 - Database side tracing with Extended Events
4. LINQ to Entities standard queries
 - Sequence queries
 - Generation: DefaultIfEmpty
 - Filtering (restriction): Where, OfType
 - Mapping (projection): Select
 - Grouping: GroupBy
 - Join
 - Inner join: Join, SelectMany, GroupJoin, Select
 - Outer join: GroupJoin, Select, SelectMany
 - Cross join and self join: SelectMany, Join
 - Concatenation: Concat
 - Set: Distinct, Union, Intersect, Except
 - Partitioning: Take, Skip
 - Ordering: OrderBy, ThenBy, OrderByDescending, ThenByDescending
 - Conversion: Cast, AsQueryable
 - Value queries
 - Element: First, FirstOrDefault, Single, SingleOrDefault
 - Aggregation: Count, LongCount, Min, Max, Sum, Average
 - Quantifier: All, Any, Contains
5. LINQ to Entities internals: Query translation implementation
 - Code to LINQ expression tree
 - IQueryable<T> and IQueryProvider
 - Standard remote queries
 - Building LINQ to Entities abstract syntax tree
 - .NET expression tree to database expression tree
 - Database query abstract syntax tree
 - Compiling LINQ expressions to database expressions
 - Compiling LINQ queries
 - Compiling .NET API calls
 - Remote API call vs. local API call
 - Compile database functions and operators
 - Database expression tree to database query language
 - SQL generator and SQL command
 - Generating SQL from database expression tree
6. Loading query data
 - Deferred execution
 - Iterator pattern
 - Lazy evaluation vs. eager evaluation
 - Explicit loading
 - Eager loading
 - Lazy loading
 - The N + 1 problem
 - Disabling lazy loading
7. Manipulating relational data: Data change and transaction
 - Repository pattern and unit of work pattern
 - Tracking entities and changes
 - Tracking entities
 - Tracking entity changes and property changes
 - Tracking relationship changes
 - Enabling and disabling tracking
 - Change data
 - Create
 - Update
 - Delete
 - Transaction
 - Transaction with connection resiliency and execution strategy
 - EF Core transaction
 - ADO.NET transaction
 - Transaction scope
8. Resolving optimistic concurrency
 - Detecting concurrent conflicts
 - Resolving concurrent conflicts
 - Retaining database values (database wins)
 - Overwriting database values (client wins)
 - Merging with database values
 - Saving changes with concurrent conflict handling
Lambda Calculus via C#: The foundation of all functional programming
1. Basics
  - Expression
    - Bound variable vs. free variable
  - Reductions
    - α-conversion (alpha-conversion)
    - β-reduction (beta-reduction)
    - η-conversion (eta-conversion)
    - Normal order
    - Applicative order
  - Function composition
    - Associativity
    - Unit
2. Church encoding: Function as boolean and logic
  - Church encoding
  - Church Boolean
  - Logical operators
  - Conversion between Church Boolean and System.Boolean
  - If
3. Church encoding: Function as numeral, arithmetic and predicate
  - Church numerals
  - Increase and decrease
  - Arithmetic operators
  - Predicate and relational operators
    - Attempt of recursion
  - Conversion between Church numeral and System.UInt32
4. Church encoding: Function as tuple and signed numeral
  - Church pair (2-tuple)
    - Tuple operators
  - N-tuple
  - Signed numeral
    - Arithmetic operators
5. Church encoding: Function as list
  - Tuple as list node
    - List operators
  - Aggregation function as list node
    - List operators
  - Model everything
6. Combinatory logic
  - Combinator
  - SKI combinator calculus
    - SKI compiler: compile lambda calculus expression to SKI calculus combinator
  - Iota combinator calculus
7. Fixed point combinator and recursion
  - Normal order fixed point combinator (Y combinator) and recursion
  - Applicative order fixed point combinator (Z combinator) and recursion
8. Undecidability of equivalence
  - Halting problem
  - Equivalence problem
Category Theory via C#: The essentials and design of LINQ
1. Basics: Category and morphism
 - Category and category laws
 - DotNet category
2. Monoid
 - Monoid and monoid laws
 - Monoid as category
3. Functor and LINQ to Functors
 - Functor and functor laws
 - Endofunctor
 - Type constructor and higher-kinded type
 - LINQ to Functors
 - Built-in IEnumerable<> functor
 - Functor pattern of LINQ
 - More LINQ to Functors
4. Natural transformation
 - Natural transformation and naturality
 - Functor Category
 - Endofunctor category
5. Bifunctor
 - Bifunctor
 - Monoidal category
6. Monoidal functor and applicative functor
 - Monoidal functor
 - IEnumeable<> monoidal functor
 - Applicative functor
 - IEnumeable<> applicative functor
 - Monoidal functor vs. applicative functor
 - More Monoidal functors and applicative functors
7. Monad and LINQ to Monads
 - Monad
 - LINQ to Monads and monad laws
 - Built-in IEnumerable<> monad
 - Monad laws and Kleisli composition
 - Kleisli category
 - Monad pattern of LINQ
 - Monad vs. monoidal/applicative functor
 - More LINQ to Monads
8. Advanced LINQ to Monads
 - IO monad
 - State monad
 - Try monad
 - Reader monad
 - Writer monad
 - Continuation monad

EntityFramework.Functions: Code First Functions for Entity Framework

Fri, 19 Jan 2024 00:00:00 GMT

EntityFramework.Functions library implements Entity Framework code first support for:

Stored procedures, with:
- single result type
- multiple result types
- output parameter
Table-valued functions, returning
entity type
complex type
Scalar-valued functions
- composable
- non-composable
Aggregate functions
Built-in functions
Niladic functions
Model defined functions

EntityFramework.Functions library works on .NET Standard with Entity Framework 6.4.0. It also works on .NET 4.0, .NET 4.5, .NET 4.6, .NET 4.7, .NET 4.8 with Entity Framework 6.1.0 and later. Entity Framework is the only dependency of this library.

It can be installed through Nuget:

dotnet add package EntityFramework.Functions

or:

Install-Package EntityFramework.Functions -DependencyVersion Highest

See:

Document: https://weblogs.asp.net/Dixin/EntityFramework.Functions
Source code
APIs
[Function]
[Parameter]
[ResultType]
FunctionConvention and FunctionConvention<TFunctions>
Examples
- Add functions to entity model
- [Stored procedure, with single result type](https://weblogs.asp.net/Dixin/EntityFramework.Functions#Stored procedure,_with_single_result_type)
- Stored procedure, with output parameter
- Stored procedure, with multiple result types
- Table-valued function
- Scalar-valued function, non-composable
- Scalar-valued function, composable
- Aggregate function
- Built-in function
- Niladic function
- Model defined function
Version history
Source code: https://github.com/Dixin/EntityFramework.Functions
Nuget package: https://www.nuget.org/packages/EntityFramework.Functions

Source code

The source can be opened and built in Visual Studio 2015.

Nuget package project is used to build the nuget package from a .nuproj. It is already included in the source.

To view the sample database, or run the unit test against the sample database, please install SQL Server 2014 LocalDB or SQL Server 2016 LocalDB.

APIs

EntityFramework.Functions library provides a few simple APIs, following the pattern of Entity Framework and LINQ to SQL.

[Function]

[Function(FunctionType type, string name)] attribute derives from DbFunctionAttribute provided in Entity Framework. It is also similar to FunctionAttribute in LINQ to SQL. When a method is tagged with [Function], it maps to a database function or stored procedure. The FunctionType parameter is an enumeration, with the following members:

StoredProcedure
TableValuedFunction
ComposableScalarValuedFunction
NonComposableScalarValuedFunction
AggregateFunction
BuiltInFunction,
NiladicFunction,
ModelDefinedFunction

Examples for each function type can be found below.

The other name parameter specifies the database function/stored procedure that is mapped to. Even when C# method name is exactly the same as the mapped database function/stored procedure, this name string still has to be provided. This is required by Entity Framework.

[Function] has 2 settable properties:

Schema: It specifies the schema of the mapped database function/stored procedure, e.g. “dbo”.
ParameterTypeSemantics: It is of ParameterTypeSemantics type provided in Entity Framework. It defines the type semantics used to resolve function overloads. ParameterTypeSemantics is an enumeration of 3 members:
AllowImplicitConversion (the default)
AllowImplicitPromotion
ExactMatchOnly

Besides general [Function] attribute, a specific attribute is also provided for each function type:

[StoredProcedure]
[TableValuedFunction]
[ComposableScalarValuedFunction]
[NonComposableScalarValuedFunction]
[AggregateFunction]
[BuiltInFunction]
[NiladicFunction]
[ModelDefinedFunction]

[Parameter]

[Parameter] tags the function parameter to specify the mapped database function/stored procedure’s parameter name and type. It is similar to ParameterAttribute in LINQ to SQL.

[Parameter] has 3 settable properties:

Name: the name of the mapped parameter in database.
DbType: the tyoe of the mapped parameter in database, like “money”
ClrType: the type of the mapping .NET parameter.
In Entity Framework, when a parameter is a output parameter, it has to be of ObjectParameter type. In this case, the mapping CLR type cannot be predicted and has to be provided by [Parameter]’s ClrType property.
In other cases, ClrType property can be omitted. At runtime, If ClrType conflicts with CLR parameter’s actual declaration CLR type, an exception will be thrown.

[Parameter] can be omitted. when:

the parameter is not an output parameter
and its name is the same as the mapped database parameter

[Parameter] can also be used to tag the return value of method, to specify the DbType of the mapped database function return value, which is also the same as LINQ to SQL. Please see examples below.

[ResultType]

[ResultType(Type type)] is exactly the same as ResultTypeAttribute in LINQ to SQL. Its constructor accepts a Type parameter to specify the return type of stored procedure. Typically, when the stored procedure has multiple result types, the mapping method can be tagged with multiple [ResultType]s.

[ResultType] cannot be used for functions.

FunctionConvention and FunctionConvention<TFunctions>

FunctionConvention and FunctionConvention<TFunctions> implements Entity Framework’s IStoreModelConvention<EntityContainer> contract. They must be added to specify in what Type the mapping methods are located.

When the functions are added to entity model, the entity types and complex types used by functions should be added to entity model too. Entity Framework does not take care of types tagged with [ComplexType], so this library provides a AddComplexTypesFromAssembly extension method for this.

For convenience, 2 extension methods AddFunctions/AddFunction<TFunctions> are provided. When they are called:

FunctionConvention/FunctionConvention<TFunctions> is added to entity model.
AddComplexTypesFromAssembly is automatically called. In the assembly of TFunction, types tagged with [ComplextType] are added to entity model.

Examples

The following examples uses Microsoft’s AdventureWorks sample database for SQL Server 2014. The database can also be found in this library’s source repository on GitHub.

Add functions to entity model

Before calling any code first function, FunctionConvention or FunctionConvention<TFunctions> must be added to DbModelBuilder of the DbContext, so are the complex types used by functions:

public partial class AdventureWorks : DbContext
{
    protected override void OnModelCreating(DbModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        // Add functions on AdventureWorks to entity model.
        modelBuilder.Conventions.Add(new FunctionConvention<AdventureWorks>());

        // Add all complex types used by functions.
        modelBuilder.ComplexType<ContactInformation>();
        modelBuilder.ComplexType<ManagerEmployee>();
        // ...
    }
}

Here new FunctionConvention<T>() is equivalent to new FunctionConvention(typeof(T)). The non-generic version is provided because in C# static class cannot be used as type argument:

public partial class AdventureWorks : DbContext
{
    protected override void OnModelCreating(DbModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        // Add functions on AdventureWorks and StaticClass to entity model.
        modelBuilder.Conventions.Add(new FunctionConvention<AdventureWorks>());
        modelBuilder.Conventions.Add(new FunctionConvention(typeof(StaticClass)));

        // Add all complex types in the assembly of AdventureWorks.
        modelBuilder.AddComplexTypesFromAssembly(typeof(AdventureWorks).Assembly);
    }
}

Also, AddFunctions/AddFunction<TFunctions> extension methods are provided as a shortcut, which automatically add all complex types in the assembly of TFunctions.

public partial class AdventureWorks : DbContext
{
    protected override void OnModelCreating(DbModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        // Add functions and complex types to model.
        modelBuilder.AddFunctions<AdventureWorks>();
        modelBuilder.AddFunctions(typeof(AdventureWorksFunctions));
        modelBuilder.AddFunctions(typeof(BuiltInFunctions));
        modelBuilder.AddFunctions(typeof(NiladicFunctions));
    }
}

Stored procedure, with single result type

The AdventureWorks database has a sample stored procedure uspGetManagerEmployees. Its return type can be viewed with dm_exec_describe_first_result_set:

SELECT *
FROM sys.dm_exec_describe_first_result_set(N'dbo.uspGetManagerEmployees', NULL, 0);

An entity type or a complex type can be defined to represent above return type:

[ComplexType]
public class ManagerEmployee
{
    public int? RecursionLevel { get; set; }

    public string OrganizationNode { get; set; }

    public string ManagerFirstName { get; set; }

    public string ManagerLastName { get; set; }

    public int? BusinessEntityID { get; set; }

    public string FirstName { get; set; }

    public string LastName { get; set; }
}

It is tagged with [ComplexType], which is provided in System.ComponentModel.DataAnnotations.dll, and used by Entity Framework. When calling AddFunctions(typeof(TFunctions))/AddFunction<TFunctions>(), types tagged with [ComplexType] in the same assembly are added to entity model too.

Now the mapping method can be defined:

public partial class AdventureWorks
{
    public const string dbo = nameof(dbo);

    // Defines stored procedure returning a single result type: 
    // - a ManagerEmployee sequence.
    [Function(FunctionType.StoredProcedure, nameof(uspGetManagerEmployees), Schema = dbo)]
    public ObjectResult<ManagerEmployee> uspGetManagerEmployees(int? BusinessEntityID)
    {
        ObjectParameter businessEntityIdParameter = BusinessEntityID.HasValue
            ? new ObjectParameter(nameof(BusinessEntityID), BusinessEntityID)
            : new ObjectParameter(nameof(BusinessEntityID), typeof(int));

        return this.ObjectContext().ExecuteFunction<ManagerEmployee>(
            nameof(this.uspGetManagerEmployees), businessEntityIdParameter);
    }
}

In its body, it should call ExecuteFunction on ObjectContext. Here ObjectContext method is an extension method provided by this library.

Then it can be called as following:

[TestMethod]
public void CallStoredProcedureWithSingleResult()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        ObjectResult<ManagerEmployee> employees = database.uspGetManagerEmployees(2);
        Assert.IsTrue(employees.Any());
    }
}

The above call is translated to the following SQL, which can be viewed with SQL Server Profiler:

exec [dbo].[uspGetManagerEmployees] @BusinessEntityID=2

Stored procedure, with output parameter

As fore mentioned, stored procedure’s output parameter is represented by ObjectParameter and must be tagged with [Parameter], with ClrType provided:

private const string uspLogError = nameof(uspLogError);

// Defines stored procedure accepting an output parameter.
// Output parameter must be ObjectParameter, with ParameterAttribute.ClrType provided.
[Function(FunctionType.StoredProcedure, uspLogError, Schema = dbo)]
public int LogError([Parameter(DbType = "int", ClrType = typeof(int))]ObjectParameter ErrorLogID) =>
    this.ObjectContext().ExecuteFunction(uspLogError, ErrorLogID);

Then it can be called as:

[TestMethod]
public void CallStoreProcedureWithOutParameter()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        ObjectParameter errorLogId = new ObjectParameter("ErrorLogID", typeof(int)) { Value = 5 };
        int? rows = database.LogError(errorLogId);
        Assert.AreEqual(0, errorLogId.Value);
        Assert.AreEqual(typeof(int), errorLogId.ParameterType);
        Assert.AreEqual(-1, rows);
    }
}

The call is translated to:

declare @p1 int
set @p1=0
exec [dbo].[uspLogError] @ErrorLogID=@p1 output
select @p1

Stored procedure, with multiple result types

The following stored procedure returns 2 different types of results: a sequence of ProductCategory row(s), and a sequence of ProductSubcategory row(s).

CREATE PROCEDURE [dbo].[uspGetCategoryAndSubCategory]
    @CategoryID int
AS
BEGIN
    SELECT [Category].[ProductCategoryID], [Category].[Name]
        FROM [Production].[ProductCategory] AS [Category] 
        WHERE [Category].[ProductCategoryID] = @CategoryID;

    SELECT [Subcategory].[ProductSubcategoryID], [Subcategory].[Name], [Subcategory].[ProductCategoryID]
        FROM [Production].[ProductSubcategory] As [Subcategory]
        WHERE [Subcategory].[ProductCategoryID] = @CategoryID;
END
GO

The involved ProductCategory table and ProductSubcategory table can be represented as:

public partial class AdventureWorks
{
    public const string Production = nameof(Production);

    public DbSet<ProductCategory> ProductCategories { get; set; }

    public DbSet<ProductSubcategory> ProductSubcategories { get; set; }
}

[Table(nameof(ProductCategory), Schema = AdventureWorks.Production)]
public partial class ProductCategory
{
    [Key]
    [DatabaseGenerated(DatabaseGeneratedOption.Identity)]
    public int ProductCategoryID { get; set; }

    [MaxLength(50)]
    public string Name { get; set; }
}

[Table(nameof(ProductSubcategory), Schema = AdventureWorks.Production)]
public partial class ProductSubcategory
{
    [Key]
    [DatabaseGenerated(DatabaseGeneratedOption.Identity)]
    public int ProductSubcategoryID { get; set; }

    [MaxLength(50)]
    public string Name { get; set; }
}

Above ProductCategory and ProductSubcategory classes are tagged with [Table], so they will be added to entity model automatically by Entity Framework.

Multiple return types can be specified by [ReturnType]. The return type defined on the method will be merged into the return types from [ReturnType]s, and be at the first position:

// Defines stored procedure returning multiple result types: 
// - a ProductCategory sequence.
// - a ProductSubcategory sequence.
[Function(FunctionType.StoredProcedure, nameof(uspGetCategoryAndSubCategory), Schema = dbo)]
[ResultType(typeof(ProductCategory))]
[ResultType(typeof(ProductSubcategory))]
public ObjectResult<ProductCategory> uspGetCategoryAndSubCategory(int CategoryID)
{
    ObjectParameter categoryIdParameter = new ObjectParameter(nameof(CategoryID), CategoryID);
    return this.ObjectContext().ExecuteFunction<ProductCategory>(
        nameof(this.uspGetCategoryAndSubCategory), categoryIdParameter);
}

Then it can be called to retrieve one ProductCategory sequence, and one ProductSubcategory sequence:

[TestMethod]
public void CallStoreProcedureWithMultipleResults()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        // The first type of result type: a sequence of ProductCategory objects.
        ObjectResult<ProductCategory> categories = database.uspGetCategoryAndSubCategory(1);
        Assert.IsNotNull(categories.Single());
        // The second type of result type: a sequence of ProductCategory objects.
        ObjectResult<ProductSubcategory> subcategories = categories.GetNextResult<ProductSubcategory>();
        Assert.IsTrue(subcategories.Any());
    }
}

The SQL translation is normal:

exec [dbo].[uspGetCategoryAndSubCategory] @CategoryID=1

Table-valued function

The AdventureWorks sample database has a table-valued function, dbo.ufnGetContactInformation, its return type can be also represented as another complex type:

[ComplexType]
public class ContactInformation
{
    public int PersonID { get; set; }

    public string FirstName { get; set; }

    public string LastName { get; set; }

    public string JobTitle { get; set; }

    public string BusinessEntityType { get; set; }
}

Then the ufnGetContactInformation function can be mapped by:

// Defines table-valued function, which must return IQueryable<T>.
[Function(FunctionType.TableValuedFunction, nameof(ufnGetContactInformation), Schema = dbo)]
public IQueryable<ContactInformation> ufnGetContactInformation(
    [Parameter(DbType = "int", Name = "PersonID")]int? personId)
{
    ObjectParameter personIdParameter = personId.HasValue
        ? new ObjectParameter("PersonID", personId)
        : new ObjectParameter("PersonID", typeof(int));

    return this.ObjectContext().CreateQuery<ContactInformation>(
        $"[{nameof(this.ufnGetContactInformation)}](@{nameof(personId)})", personIdParameter);
}

Its return type should be IQueryable<T>, so that it is composable in LINQ to Entities. And it can be called:

[TestMethod]
public void CallTableValuedFunction()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        IQueryable<ContactInformation> employees = database.ufnGetContactInformation(1).Take(2);
        Assert.IsNotNull(employees.Single());
    }
}

The above ufnGetContactInformation call and Take call will be translated to one single SQL query:

exec sp_executesql N'SELECT TOP (2) 
    [top].[C1] AS [C1], 
    [top].[PersonID] AS [PersonID], 
    [top].[FirstName] AS [FirstName], 
    [top].[LastName] AS [LastName], 
    [top].[JobTitle] AS [JobTitle], 
    [top].[BusinessEntityType] AS [BusinessEntityType]
    FROM ( SELECT TOP (2) 
        [Extent1].[PersonID] AS [PersonID], 
        [Extent1].[FirstName] AS [FirstName], 
        [Extent1].[LastName] AS [LastName], 
        [Extent1].[JobTitle] AS [JobTitle], 
        [Extent1].[BusinessEntityType] AS [BusinessEntityType], 
        1 AS [C1]
        FROM [dbo].[ufnGetContactInformation](@PersonID) AS [Extent1]
    )  AS [top]',N'@PersonID int',@PersonID=1

Scalar-valued function, non-composable

For scalar-valued function. the return value becomes a primitive non-collection type.

// Defines scalar-valued function (non-composable), 
// which cannot be used in LINQ to Entities queries;
// and can be called directly.
[Function(FunctionType.NonComposableScalarValuedFunction, nameof(ufnGetProductStandardCost), Schema = dbo)]
[return: Parameter(DbType = "money")]
public decimal? ufnGetProductStandardCost(
    [Parameter(DbType = "int")]int ProductID,
    [Parameter(DbType = "datetime")]DateTime OrderDate)
{
    ObjectParameter productIdParameter = new ObjectParameter(nameof(ProductID), ProductID);
    ObjectParameter orderDateParameter = new ObjectParameter(nameof(OrderDate), OrderDate);
    return this.ObjectContext().ExecuteFunction<decimal?>(
        nameof(this.ufnGetProductStandardCost), productIdParameter, orderDateParameter).SingleOrDefault();
}

In this case, [Parameter] can tag its return type.

It can be called directly just like other above methods:

[TestMethod]
public void CallNonComposableScalarValuedFunction()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        decimal? cost = database.ufnGetProductStandardCost(999, DateTime.Now);
        Assert.IsNotNull(cost);
        Assert.IsTrue(cost > 1);
    }
}

And the translated SQL is:

exec sp_executesql N'SELECT [dbo].[ufnGetProductStandardCost](@ProductID, @OrderDate)',N'@ProductID int,@OrderDate datetime2(7)',@ProductID=999,@OrderDate='2015-12-28 02:22:53.0353800'

However, since it is specified to be non-composable, it cannot be translated by Entity Framework in LINQ to Entities queries:

[TestMethod]
public void NonComposableScalarValuedFunctionInLinq()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        try
        {
            database
                .Products
                .Where(product => product.ListPrice >= database.ufnGetProductStandardCost(999, DateTime.Now))
                .ToArray();
            Assert.Fail();
        }
        catch (NotSupportedException)
        {
        }
    }
}

This is by design of Entity Framework.

Scalar-valued function, composable

The composable scalar-valued function is very similar:

// Defines scalar-valued function (composable),
// which can only be used in LINQ to Entities queries, where its body will never be executed;
// and cannot be called directly.
[Function(FunctionType.ComposableScalarValuedFunction, nameof(ufnGetProductListPrice), Schema = dbo)]
[return: Parameter(DbType = "money")]
public decimal? ufnGetProductListPrice(
    [Parameter(DbType = "int")] int ProductID,
    [Parameter(DbType = "datetime")] DateTime OrderDate) => 
        Function.CallNotSupported<decimal?>();

The difference is, it works in LINQ to Entities queries, but cannot be called directly. As a result, its body will never be executed. So in the body, it can just throw an exception. This library provides a Function.,CallNotSupported help methods for convenience, which just throws a NotSupportedException.

[TestMethod]
public void ComposableScalarValuedFunctionInLinq()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        IQueryable<Product> products = database
            .Products
            .Where(product => product.ListPrice <= database.ufnGetProductListPrice(999, DateTime.Now));
        Assert.IsTrue(products.Any());
    }
}

[TestMethod]
public void CallComposableScalarValuedFunction()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        try
        {
            database.ufnGetProductListPrice(999, DateTime.Now);
            Assert.Fail();
        }
        catch (NotSupportedException)
        {
        }
    }
}

The above LINQ query, containing composable scalar-valued function, is translated to:

SELECT 
    CASE WHEN ( EXISTS (SELECT 
        1 AS [C1]
        FROM [Production].[Product] AS [Extent1]
        WHERE [Extent1].[ListPrice] <= ([dbo].[ufnGetProductListPrice](999, SysDateTime()))
    )) THEN cast(1 as bit) WHEN ( NOT EXISTS (SELECT 
        1 AS [C1]
        FROM [Production].[Product] AS [Extent2]
        WHERE [Extent2].[ListPrice] <= ([dbo].[ufnGetProductListPrice](999, SysDateTime()))
    )) THEN cast(0 as bit) END AS [C1]
    FROM  ( SELECT 1 AS X ) AS [SingleRowTable1]

Aggregate function

To demonstrate the mapping of user defined aggregate, the following aggregate function can be defined:

[Serializable]
[SqlUserDefinedAggregate(
    Format.UserDefined,
    IsInvariantToNulls = true,
    IsInvariantToDuplicates = false,
    IsInvariantToOrder = false,
    MaxByteSize = 8000)]
public class Concat : IBinarySerialize
{
    private const string Separator = ", ";

    private StringBuilder concat;

    public void Init()
    {
    }

    public void Accumulate(SqlString sqlString) => this.concat = this.concat?
        .Append(Separator).Append(sqlString.IsNull ? null : sqlString.Value)
        ?? new StringBuilder(sqlString.IsNull ? null : sqlString.Value);

    public void Merge(Concat concat) => this.concat.Append(concat.concat);

    public SqlString Terminate() => new SqlString(this.concat?.ToString());

    public void Read(BinaryReader reader) => this.concat = new StringBuilder(reader.ReadString());

    public void Write(BinaryWriter writer) => writer.Write(this.concat?.ToString() ?? string.Empty);
}

Concat takes 1 parameter, just like COUNT(), SUM(), etc. The following ConcatWith aggregate function accepts 2 parameters, a value and a separator:

[Serializable]
[SqlUserDefinedAggregate(
    Format.UserDefined,
    IsInvariantToNulls = true,
    IsInvariantToDuplicates = false,
    IsInvariantToOrder = false,
    MaxByteSize = 8000)]
public class ConcatWith : IBinarySerialize
{
    private StringBuilder concatWith;

    public void Init()
    {
    }

    public void Accumulate(SqlString sqlString, SqlString separator) => this.concatWith = this.concatWith?
        .Append(separator.IsNull ? null : separator.Value)
        .Append(sqlString.IsNull ? null : sqlString.Value)
        ?? new StringBuilder(sqlString.IsNull ? null : sqlString.Value);

    public void Merge(ConcatWith concatWith) => this.concatWith.Append(concatWith.concatWith);

    public SqlString Terminate() => new SqlString(this.concatWith?.ToString());

    public void Read(BinaryReader reader) => this.concatWith = new StringBuilder(reader.ReadString());

    public void Write(BinaryWriter writer) => writer.Write(this.concatWith?.ToString() ?? string.Empty);
}

Build these 2 classes into a .NET assembly, and add to database:

-- Create assembly.
CREATE ASSEMBLY [Dixin.Sql] 
FROM N'D:\OneDrive\Works\Drafts\CodeSnippets\Dixin.Sql\bin\Debug\Dixin.Sql.dll';
GO

-- Create aggregate from assembly.
CREATE AGGREGATE [Concat] (@value nvarchar(4000)) RETURNS nvarchar(max)
EXTERNAL NAME [Dixin.Sql].[Dixin.Sql.Concat];
GO

CREATE AGGREGATE [ConcatWith] (@value nvarchar(4000), @separator nvarchar(40)) RETURNS nvarchar(max)
EXTERNAL NAME [Dixin.Sql].[Dixin.Sql.ConcatWith];
GO

Now Concat and ConcatWith can be used in SQL:

SELECT [Subcategory].[ProductCategoryID], COUNT([Subcategory].[Name]), [dbo].[Concat]([Subcategory].[Name])
FROM [Production].[ProductSubcategory] AS [Subcategory]
GROUP BY [Subcategory].[ProductCategoryID];

SELECT [dbo].[Concat](Name) FROM Production.ProductCategory;

SELECT [Subcategory].[ProductCategoryID], COUNT([Subcategory].[Name]), [dbo].[ConcatWith]([Subcategory].[Name], N' | ')
FROM [Production].[ProductSubcategory] AS [Subcategory]
GROUP BY [Subcategory].[ProductCategoryID];

SELECT [dbo].[ConcatWith](Name, N' | ') FROM Production.ProductCategory;

To map them in C#, the following methods can be defined:

public static class AdventureWorksFunctions
{
    // Defines aggregate function, which must have one singele IEnumerable<T> or IQueryable<T> parameter.
    // It can only be used in LINQ to Entities queries, where its body will never be executed;
    // and cannot be called directly.
    [Function(FunctionType.AggregateFunction, nameof(Concat), Schema = AdventureWorks.dbo)]
    public static string Concat(this IEnumerable<string> value) => Function.CallNotSupported<string>();

    // Aggregate function with more than more parameter is not supported by Entity Framework.
    // The following cannot to translated in LINQ queries.
    // [Function(FunctionType.AggregateFunction, nameof(ConcatWith), Schema = AdventureWorks.dbo)]
    // public static string ConcatWith(this IEnumerable<string> value, string separator) => 
    //    Function.CallNotSupported<string>();
}

Apparently, aggregate functions cannot be called directly, so their bodies just throw exception. Unfortunately, above ConcatWith cannot be translated, because currently Entity Framework does not support aggregate function with more than one parameters.

They are defined as extension methods of IEnumerable<T>, so that they can easily be used in LINQ to :

[TestMethod]
public void AggregateFunctionInLinq()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        var categories = database.ProductSubcategories
            .GroupBy(subcategory => subcategory.ProductCategoryID)
            .Select(category => new
            {
                CategoryId = category.Key,
                SubcategoryNames = category.Select(subcategory => subcategory.Name).Concat()
            })
            .ToArray();
        Assert.IsTrue(categories.Length > 0);
        categories.ForEach(category =>
            {
                Assert.IsTrue(category.CategoryId > 0);
                Assert.IsFalse(string.IsNullOrWhiteSpace(category.SubcategoryNames));
            });
    }
}

Above query will be translated to SQL with Concat call:

SELECT 
    1 AS [C1], 
    [GroupBy1].[K1] AS [ProductCategoryID], 
    [GroupBy1].[A1] AS [C2]
    FROM ( SELECT 
        [Extent1].[ProductCategoryID] AS [K1], 
        [dbo].[Concat]([Extent1].[Name]) AS [A1]
        FROM [Production].[ProductSubcategory] AS [Extent1]
        GROUP BY [Extent1].[ProductCategoryID]
    )  AS [GroupBy1]

The reason is Entity Framework does not support aggregate function with more than one parameters.

Built-in function

SQL Server provides a lot of built-in functions. They can be easily represented with [Function] tag. Take LEFT function as example:

It is a string function, returns the left part of a string with the specified number of characters. So, in C#, just defines a function accepting a string parameter and a int parameter, and returns a string:

public static class BuiltInFunctions
{
    [Function(FunctionType.BuiltInFunction, "LEFT")]
    public static string Left(this string value, int count) => Function.CallNotSupported<string>();
}

Again, it can only be used in LINQ to Entities and cannot be called directly. So in its body, it just simply throw an exception. It is implemented as an extension method of string, for convenience.

[TestMethod]
public void BuitInFunctionInLinq()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        var categories = database.ProductSubcategories
            .GroupBy(subcategory => subcategory.ProductCategoryID)
            .Select(category => new
            {
                CategoryId = category.Key,
                SubcategoryNames = category.Select(subcategory => subcategory.Name.Left(4)).Concat()
            })
            .ToArray();
        Assert.IsTrue(categories.Length > 0);
        categories.ForEach(category =>
        {
            Assert.IsTrue(category.CategoryId > 0);
            Assert.IsFalse(string.IsNullOrWhiteSpace(category.SubcategoryNames));
        });
    }
}

The above query is translated to SQL with LEFT call:

SELECT 
    1 AS [C1], 
    [GroupBy1].[K1] AS [ProductCategoryID], 
    [GroupBy1].[A1] AS [C2]
    FROM ( SELECT 
        [Extent1].[K1] AS [K1], 
        [dbo].[Concat]([Extent1].[A1]) AS [A1]
        FROM ( SELECT 
            [Extent1].[ProductCategoryID] AS [K1], 
            LEFT([Extent1].[Name], 4) AS [A1]
            FROM [Production].[ProductSubcategory] AS [Extent1]
        )  AS [Extent1]
        GROUP BY [K1]
    )  AS [GroupBy1]

Niladic function

Niladic functions are functions called without parentheses, e.g., these SQL-92 niladic functions:

CURRENT_TIMESTAMP
CURRENT_USER
SESSION_USER
USER

In C#:

public static class NiladicFunctions
{
    [Function(FunctionType.NiladicFunction, "CURRENT_TIMESTAMP")]
    public static DateTime? CurrentTimestamp() => Function.CallNotSupported<DateTime?>();

    [Function(FunctionType.NiladicFunction, "CURRENT_USER")]
    public static string CurrentUser() => Function.CallNotSupported<string>();

    [Function(FunctionType.NiladicFunction, "SESSION_USER")]
    public static string SessionUser() => Function.CallNotSupported<string>();

    [Function(FunctionType.NiladicFunction, "SYSTEM_USER")]
    public static string SystemUser() => Function.CallNotSupported<string>();

    [Function(FunctionType.NiladicFunction, "USER")]
    public static string User() => Function.CallNotSupported<string>();
}

When they are called:

[TestMethod]
public void NiladicFunctionInLinq()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        var firstCategory = database.ProductSubcategories
            .GroupBy(subcategory => subcategory.ProductCategoryID)
            .Select(category => new
            {
                CategoryId = category.Key,
                SubcategoryNames = category.Select(subcategory => subcategory.Name.Left(4)).Concat(),
                CurrentTimestamp = NiladicFunctions.CurrentTimestamp(),
                CurrentUser = NiladicFunctions.CurrentUser(),
                SessionUser = NiladicFunctions.SessionUser(),
                SystemUser = NiladicFunctions.SystemUser(),
                User = NiladicFunctions.User()
            })
            .First();
        Assert.IsNotNull(firstCategory);
        Assert.IsNotNull(firstCategory.CurrentTimestamp);
        Assert.IsTrue(DateTime.Now >= firstCategory.CurrentTimestamp);
        Assert.AreEqual("dbo", firstCategory.CurrentUser, true, CultureInfo.InvariantCulture);
        Assert.AreEqual("dbo", firstCategory.SessionUser, true, CultureInfo.InvariantCulture);
        Assert.AreEqual($@"{Environment.UserDomainName}\{Environment.UserName}", firstCategory.SystemUser, true, CultureInfo.InvariantCulture);
        Assert.AreEqual("dbo", firstCategory.User, true, CultureInfo.InvariantCulture);
    }
}

They are translated to SQL calls without parentheses:

SELECT 
    [Limit1].[C2] AS [C1], 
    [Limit1].[ProductCategoryID] AS [ProductCategoryID], 
    [Limit1].[C1] AS [C2], 
    [Limit1].[C3] AS [C3], 
    [Limit1].[C4] AS [C4], 
    [Limit1].[C5] AS [C5], 
    [Limit1].[C6] AS [C6], 
    [Limit1].[C7] AS [C7]
    FROM ( SELECT TOP (1) 
        [GroupBy1].[A1] AS [C1], 
        [GroupBy1].[K1] AS [ProductCategoryID], 
        1 AS [C2], 
        CURRENT_TIMESTAMP AS [C3], 
        CURRENT_USER AS [C4], 
        SESSION_USER AS [C5], 
        SYSTEM_USER AS [C6], 
        USER AS [C7]
        FROM ( SELECT 
            [Extent1].[K1] AS [K1], 
            [dbo].[Concat]([Extent1].[A1]) AS [A1]
            FROM ( SELECT 
                [Extent1].[ProductCategoryID] AS [K1], 
                LEFT([Extent1].[Name], 4) AS [A1]
                FROM [Production].[ProductSubcategory] AS [Extent1]
            )  AS [Extent1]
            GROUP BY [K1]
        )  AS [GroupBy1]
    )  AS [Limit1]

Model defined function

The following code defines a FormatName function for the Person model:

public static class ModelDefinedFunctions
{
    [ModelDefinedFunction(nameof(FormatName), "EntityFramework.Functions.Tests.Examples",
        @"(CASE 
            WHEN [Person].[Title] IS NOT NULL
            THEN [Person].[Title] + N' ' 
            ELSE N'' 
        END) + [Person].[FirstName] + N' ' + [Person].[LastName]")]
    public static string FormatName(this Person person) =>
        $"{(person.Title == null ? string.Empty : person.Title + " ")}{person.FirstName} {person.LastName}";

    [ModelDefinedFunction(nameof(ParseDecimal), "EntityFramework.Functions.Tests.Examples", "cast([Person].[BusinessEntityID] as Decimal(20,8))")]
    public static decimal ParseDecimal(this Person person) => Convert.ToDecimal(person.BusinessEntityID);
}

When FormatName is called in LINQ to Entities query:

[TestMethod]
public void ModelDefinedFunctionInLinqTest()
{
    using (AdventureWorks database = new AdventureWorks())
    {
        var employees = from employee in database.Persons
                        where employee.Title != null
                        let formatted = employee.FormatName()
                        select new
                        {
                            formatted,
                            employee
                        };
        var employeeData = employees.Take(1).ToList().FirstOrDefault();
        Assert.IsNotNull(employeeData);
        Assert.IsNotNull(employeeData.formatted);
        Assert.AreEqual(employeeData.employee.FormatName(), employeeData.formatted);
    }

    using (AdventureWorks database = new AdventureWorks())
    {
        var employees = from employee in database.Persons
                        where employee.Title != null
                        select new
                        {
                            Decimal = employee.ParseDecimal(),
                            Int32 = employee.BusinessEntityID
                        };
        var employeeData = employees.Take(1).ToList().FirstOrDefault();
        Assert.IsNotNull(employeeData);
        Assert.AreEqual(employeeData.Decimal, Convert.ToInt32(employeeData.Int32));
    }
}

The queries are translated to:

SELECT 
    [Limit1].[BusinessEntityID] AS [BusinessEntityID], 
    [Limit1].[C1] AS [C1], 
    [Limit1].[Title] AS [Title], 
    [Limit1].[FirstName] AS [FirstName], 
    [Limit1].[LastName] AS [LastName]
    FROM ( SELECT TOP (1) 
        [Extent1].[BusinessEntityID] AS [BusinessEntityID], 
        [Extent1].[Title] AS [Title], 
        [Extent1].[FirstName] AS [FirstName], 
        [Extent1].[LastName] AS [LastName], 
        CASE WHEN ([Extent1].[Title] IS NOT NULL) THEN [Extent1].[Title] + N' ' ELSE N'' END + [Extent1].[FirstName] + N' ' + [Extent1].[LastName] AS [C1]
        FROM [Person].[Person] AS [Extent1]
        WHERE [Extent1].[Title] IS NOT NULL
    )  AS [Limit1]

SELECT 
    [Limit1].[BusinessEntityID] AS [BusinessEntityID], 
    [Limit1].[C1] AS [C1]
    FROM ( SELECT TOP (1) 
        [Extent1].[BusinessEntityID] AS [BusinessEntityID], 
         CAST( [Extent1].[BusinessEntityID] AS decimal(20,8)) AS [C1]
        FROM [Person].[Person] AS [Extent1]
        WHERE [Extent1].[Title] IS NOT NULL
    )  AS [Limit1]

Version history

This library adopts the http://semver.org standard for semantic versioning.

1.0.0: Initial release.
1.0.1: Bug fix.
1.1.0: Bug fix, and shortcut APIs for each function type:
[StoredProcedure]
[TableValuedFunction]
[ComposableScalarValuedFunction]
[NonComposableScalarValuedFunction]
[AggregateFunction]
[BuiltInFunction]
[NiladicFunction]
1.2.0: Support model defined function with [ModelDefinedFunction].
1.3.0: Support entity type and complex type defined in different assembly/namespace. Support table-valued function returning entity type or complex type.
1.3.1: Fix a regression causing complex type not working properly with PostgreSQL.
1.4.0: Sign assembly with strong named key. Fix minor issues.
1.5.0: Support .NET Standard.

C# 10 new feature CallerArgumentExpression, argument check and more

Sun, 14 Nov 2021 00:00:00 GMT

The CallerArgumentExpression has been discussed for years, it was supposed to a part of C# 8.0 but got delayed. Finally this month it is delivered along with C# 10 and .NET 6.

CallerArgumentExpressionAttribute and argument compilation

In C# 10, [CallerArgumentExpression(parameterName)] can be used to direct the compiler to capture the specified argument’s expression as text. For example:

using System.Runtime.CompilerServices;

void Function(int a, TimeSpan b, [CallerArgumentExpression("a")] string c = "", [CallerArgumentExpression("b")] string d = "")
{
    Console.WriteLine($"Called with value {a} from expression '{c}'");
    Console.WriteLine($"Called with value {b} from expression '{d}'");
}

When calling above function, The magic happens at compile time:

Function(1, default);
// Compiled to: 
Function(1, default, "1", "default");

int x = 1;
TimeSpan y = TimeSpan.Zero;
Function(x, y);
// Compiled to:
Function(x, y, "x", "y");

Function(int.Parse("2") + 1 + Math.Max(2, 3), TimeSpan.Zero - TimeSpan.MaxValue);
// Compiled to:
Function(int.Parse("2") + 1 + Math.Max(2, 3), TimeSpan.Zero - TimeSpan.MaxValue, "int.Parse(\"2\") + 1 + Math.Max(2, 3)", "TimeSpan.Zero - TimeSpan.MaxValue");

Function’s parameter c is decorated with [CallerArgumentExpression("a")]. So when calling Function, C# compiler will pickup whatever expression passed to a, and use that expression’s text for c. Similarly, whatever expression is used for b, that expression’s text is used for d.

Argument check

The most useful scenario of this feature is argument check. In the past, a lot of argument check utility methods are created like this:

public static partial class Argument
{
    public static void NotNull<T>([NotNull] T? value, string name) where T : class
    {
        if (value is null)
        {
            throw new ArgumentNullException(name);
        }
    }

    public static void NotNullOrWhiteSpace([NotNull] string? value, string name)
    {
        if (string.IsNullOrWhiteSpace(value))
        {
            throw new ArgumentException(string.Format(CultureInfo.CurrentCulture, Resources.StringCannotBeEmpty, name));
        }
    }

    public static void NotNegative(int value, string name)
    {
        if (value < 0)
        {
            throw new ArgumentOutOfRangeException(name, value, string.Format(CultureInfo.CurrentCulture, Resources.ArgumentCannotBeNegative, name));
        }
    }
}

So they can be used as:

public partial record Person
{
    public Person(string name, int age, Uri link)
    {
        Argument.NotNullOrWhiteSpace(name, nameof(name));
        Argument.NotNegative(age, nameof(age));
        Argument.NotNull(link, nameof(link));

        this.Name = name;
        this.Age = age;
        this.Link = link.ToString();
    }

    public string Name { get; }
    public int Age { get; }
    public string Link { get; }
}

The problem is, it is very annoying to pass argument name every time. There are some ways to get rid of manually passing argument name, but these approaches introduces other issues. For example, a lambda expression with closure can be used:

public partial record Person
{
    public Person(Uri link)
    {
        Argument.NotNull(() => link);

        this.Link = link.ToString();
    }
}

And this version of NotNull can take a function:

public static partial class Argument
{
    public static void NotNull<T>(Func<T> value)
    {
        if (value() is null)
        {
            throw new ArgumentNullException(GetName(value));
        }
    }

    private static string GetName<T>(Func<T> func)
    {
        // func: () => arg is compiled to DisplayClass with a field and a method. That method is func.
        object displayClassInstance = func.Target!;
        FieldInfo closure = displayClassInstance.GetType()
            .GetFields(BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Instance)
            .Single();
        return closure.Name;
    }
}

See my post on what is closure and how C# compiles closure.

Lambda expression can be also compiled to expression tree. So NotNull can be implemented to take an expression too (See my post on what is expression tree and how C# compiles expression tree):

public static partial class Argument
{
    public static void NotNull<T>(Expression<Func<T>> value)
    {
        if (value.Compile().Invoke() is null)
        {
            throw new ArgumentNullException(GetName(value));
        }
    }

    private static string GetName<T>(Expression<Func<T>> expression)
    {
        // expression: () => arg is compiled to DisplayClass with a field. Here expression body is to access DisplayClass instance's field.
        MemberExpression displayClassInstance = (MemberExpression)expression.Body;
        MemberInfo closure = displayClassInstance.Member;
        return closure.Name;
    }
}

These approaches introduce the lambda syntax and performance overhead at runtime. And they are extremely fragile too. Now C# 10’s CallerArgumentExpression finally provides a cleaner solution:

public static partial class Argument
{
    public static T NotNull<T>([NotNull] this T? value, [CallerArgumentExpression("value")] string name = "")
        where T : class =>
        value is null ? throw new ArgumentNullException(name) : value;

    public static string NotNullOrWhiteSpace([NotNull] this string? value, [CallerArgumentExpression("value")] string name = "") =>
        string.IsNullOrWhiteSpace(value)
            ? throw new ArgumentException(string.Format(CultureInfo.CurrentCulture, Resources.StringCannotBeEmpty, name), name)
            : value;

    public static int NotNegative(this int value, [CallerArgumentExpression("value")] string name = "") =>
        value < 0
            ? throw new ArgumentOutOfRangeException(name, value, string.Format(CultureInfo.CurrentCulture, Resources.ArgumentCannotBeNegative, name))
            : value;
}

Now the argument check can be shorter and fluent:

public record Person
{
    public Person(string name, int age, Uri link) => 
        (this.Name, this.Age, this.Link) = (name.NotNullOrWhiteSpace(), age.NotNegative(), link.NotNull().ToString());
        // Compiled to:
        // this.Name = Argument.NotNullOrWhiteSpace(name, "name");
        // this.Age = Argument.NotNegative(age, "age");
        // this.Link = Argument.NotNull(link, "link").ToString();

    public string Name { get; }
    public int Age { get; }
    public string Link { get; }
}

The argument name is generated at compile time and there is no performance overhead at runtime at all.

Assertion and logging

The other useful scenarios could be assertion and logging:

[Conditional("DEBUG")]
static void Assert(bool condition, [CallerArgumentExpression("condition")] string expression = "")
{
    if (!condition)
    {
        Environment.FailFast($"'{expression}' is false and should be true.");
    }
}

Assert(y > TimeSpan.Zero);
// Compiled to:
Assert(y > TimeSpan.Zero, "y > TimeSpan.Zero");

[Conditional("DEBUG")]
static void Log<T>(T value, [CallerArgumentExpression("value")] string expression = "")
{
    Trace.WriteLine($"'{expression}' has value '{value}'");
}

Log(Math.Min(Environment.ProcessorCount, x));
// Compiled to:
Log(Math.Min(Environment.ProcessorCount, x), "Math.Min(Environment.ProcessorCount, x)");

Use for older projects

If .NET 6.0 SDK is installed, C# 10 is available, where CallerArgumentExpression can be used targeting to .NET 5 and .NET 6. For older project targeting older .NET or .NET Standard, CallerArgumentExpressionAttribute is not available. Fortunately C# 10 and this feature can still be used with them, as long as .NET 6.0 SDK is installed. Just manually add the CallerArgumentExpressionAttribute class to your project and use it like the built-in attribute:

#if !NET5_0 && !NET6_0
namespace System.Runtime.CompilerServices;

/// <summary>
/// Allows capturing of the expressions passed to a method.
/// </summary>
[AttributeUsage(AttributeTargets.Parameter, AllowMultiple = false, Inherited = false)]
internal sealed class CallerArgumentExpressionAttribute : Attribute
{
    /// <summary>
    /// Initializes a new instance of the <see cref="T:System.Runtime.CompilerServices.CallerArgumentExpressionAttribute" /> class.
    /// </summary>
    /// <param name="parameterName">The name of the targeted parameter.</param>
    public CallerArgumentExpressionAttribute(string parameterName) => this.ParameterName = parameterName;

    /// <summary>
    /// Gets the target parameter name of the <c>CallerArgumentExpression</c>.
    /// </summary>
    /// <returns>
    /// The name of the targeted parameter of the <c>CallerArgumentExpression</c>.
    /// </returns>
    public string ParameterName { get; }
}
#endif

It should be internal so that when this assembly is referenced by another assembly, there won’t conflict with the built-in version of [CallerArgumentExpression]. Then C# 10‘s compiler will pick it up and the above magic will happen.

Windows 10 and 11 minimal setup for HDR video playback and streaming

Sun, 14 Nov 2021 00:00:00 GMT

On last Black Friday, I purchased a 50-inch 4K HDR10 smart TV with only $150. I use it as monitor for my computer. I didn’t find a walk through tutorial for the whole HDR (High Dynamic Range) setup, so here I am sharing the steps.

Prerequisites

Hardware:

Graphic card must support HDR. For example:
Nvidia GeForce GTX 900, 10 series, 16 series and RTX 20 series, 30 series
AMD Polaris Series, Vega series, Big Navi series
Monitor or TV must have the HDR feature, including DP 1.4 or HDMI 2.0 support. Other fancy features, like 10 bit color, FreeSync, G-Sync, etc., are not relevant to HDR.
Video cable must be DP 1.4 or HDMI 2.0 or higher.

Software:

Windows 11 or Windows 10 version 1803 or later.
Video player must support HDR.
Video file or stream must be HDR encoded.

Setup TV/monitor

Make sure the TV/monitor’s HDMI 2.0/DP 1.4 mode is enabled. “Auto” does not work on my TV. Switching to 2.0 makes it work.

Windows and graphic card configuration

Installing the latest Windows 10 build version 20H2, switch the default power plan from Balanced to High Performance.

Install the latest graphic card driver, switch Output dynamic range from Limited to Full.

And switch to Dynamic range from Limited to Full:

Setup video player and filter

For player, there are many options, like MPC-HC, MPC-BE, MPV, Pot Player, etc. I use MPC-HC (Media Player Classic Home Cinema), because I am used to it for more than a decade. And it is open source https://github.com/clsid2/mpc-hc. It can be easily installed via winget:

winget install –id clsid2.mpc-hc

After trying many things around, I found MadVR works the best and easist for playing normal and HDR videos. It can be installed from official website http://madvr.com/, or from Chocolatey:

choco install madvr

Launch video player, in the options, go to Playback –> Output, for “DirectShow Video” select “madVR” from the dropdown list.

Then Go to Internal Filters, click “Video decoder” button, the Properties panel pops up. For “Hardware Decoder to use”, select whatever works for your machine. For my computer, “DXVA2 (copy-back)” and “NVIDIA CUVID” both work.

Then launch madVR settings. You can go to its installation folder (e.g. C:\ProgramData\chocolatey\lib\madvr\tools), launch madHcCtrl.exe. It is luanched and minimized to the system tray. Right click the system tray icon, click “Edit madVR settings” in the menu.

FIrst, go to devices, find your monitor, and select the correct device type:

Then go to devices –> Your monitor –> properties, select 0-255 for the level, and select the correct color depth.

Now you should be able to play HDR video files.

This article documents the minimal setup for HDR on Windows 10 and Windows 11. There are tons of more madVR settings to tweak if you have time and interest. Please see the following tools and tutorials:

Setup and use CUDA and TensorFlow in Windows Subsystem for Linux 2

Thu, 17 Dec 2020 00:00:00 GMT

Table of contents

Install Windows preview
Install WSL 2 preview
Install Nvidia driver preview and CUDA toolkit
Run CUDA sample application
Install Docker and Nvidia container toolkit
Run CUDA containers
Troubleshoot
Run WSL + CUDA + Docker + Jupyter + TensorFlow
Encoding and decoding video with GPU in WSL?

GPU support is the most requested feature in Windows Subsystem for Linux (WSL). It is available in WSL 2.0 through Windows Insiders program. And Nvidia CUDA is supported. The following diagram shows the WDDM model supporting CUDA user mode driver running inside Linux guest:

So the popular Linux AI frameworks like TensorFlow, PyTorch, etc. can work with WSL with CUDA acceleration:

This article walks through the installation of Windows, WSL, CUDA in WSL, and Docker in WSL.

Install Windows preview

First, you must enable “Optional diagnostic data”, otherwise Windows cannot join Windows Insiders.

Then, join Windows Insiders program with Microsoft account (an account can be created if you don’t have one: https://insider.windows.com/). The channel must be Dev:

Then run Windows Update. It will download the pre=release installer. Windows will restart and reinstall.

Install WSL 2 preview

In Windows, make sure the following Windows features are enabled:

WSL: dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
Virtual machine platform: dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Now restart Windows, then Windows will have WSL and the wsl command line tool. Run Windows Update again to get the latest WSL 2. When this is done, in the update history, it must show 4.19.121 or later:

Then manually install this patch: https://wslstorestorage.blob.core.windows.net/wslblob/wsl_update_x64.msi. And then run the following command as administrator to update the kernel to the latest version:

C:\WINDOWS\system32>wsl --update Checking for updates... Downloading updates... Installing updates... This change will take effect on the next full restart of WSL. To force a restart, please run 'wsl --shutdown'. Kernel version: 5.4.72

Now go to Microsoft Store, install a Linux distribution. I installed the first one Ubuntu, it is the same as the third item Ubuntu 20.04 LTS

When the installation is done, use the following command to set the default version to 2 and verify:

C:\WINDOWS\system32>wsl --set-default-version 2 For information on key differences with WSL 2 please visit https://aka.ms/wsl2 The operation completed successfully.

C:\WINDOWS\system32>wsl --list --verbose NAME STATE VERSION * Ubuntu Stopped 2 docker-desktop Stopped 2 docker-desktop-data Stopped 2

WSL version can also be set for a specific distribution:

C:\WINDOWS\system32>wsl --set-version ubuntu 2 Conversion in progress, this may take a few minutes... For information on key differences with WSL 2 please visit https://aka.ms/wsl2 The distribution is already the requested version.

Now launch the installed Ubuntu (default) from the Store, or using the command wsl:

For the first time launch, it asks for initializing the user name and password.

Install Nvidia driver preview and CUDA toolkit

In Windows, download the preview driver from Nvidia: https://developer.nvidia.com/cuda/wsl/download and install.

Then launch WSL, install CUDA:

apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub sh -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list' apt update apt install -y cuda-toolkit-11-1

Run CUDA sample application

The CUDA sample source code can be downloaded from GitHub. In WSL, run:

git clone https://github.com/NVIDIA/cuda-samples.git

Then build and run a sample application

cd cuda-samples/Samples/concurrentKernels make ./concurrentKernels

If everything is successful, it should find the GPU and show the test output:

[./concurrentKernels] - Starting... GPU Device 0: "Turing" with compute capability 7.5 > Detected Compute SM 7.5 hardware with 20 multi-processors Expected time for serial execution of 8 kernels = 0.080s Expected time for concurrent execution of 8 kernels = 0.010s Measured time for sample = 0.012s Test passed

Install Docker and Nvidia container toolkit

First, if Docker Desktop is installed in Windows, turn off the WSL integration for WSL in the distro, because It does not work with CUDA in WSL:

Now go to WSL, install Docker from there:

curl https://get.docker.com | sh

Then install Nvidia container toolkit in WSL:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list curl -s -L https://nvidia.github.io/libnvidia-container/experimental/$distribution/libnvidia-container-experimental.list | sudo tee /etc/apt/sources.list.d/libnvidia-container-experimental.list sudo apt update sudo apt install -y nvidia-docker2

When it is done, restart docker in WSL:

sudo service docker stop sudo service docker start

Run CUDA containers

Now it is ready to run CUDA container:

docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

It should give output like this:

GPU Device 0: "GeForce GTX 1650 SUPER" with compute capability 7.5 > Compute 7.5 CUDA device: [GeForce GTX 1650 SUPER] 20480 bodies, total time for 10 iterations: 30.007 ms = 139.776 billion interactions per second = 2795.517 single-precision GFLOP/s at 20 flops per interaction

Troubleshoot

Here Windows, WSL 2, Nvidia driver are all in preview. Sometimes when you run container, docker gives error:

docker: Error response from daemon: cgroups: cannot find cgroup mount destination: unknown. ERRO[0000] error waiting for container: context canceled

Then WSL has to be restarted. In Windows command prompt:

wsl –shutdown wsl

If Docker gives the following error:

Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

Then install nvidia-container-toolkit:

sudo apt install -y nvidia-container-toolkit

If Docker gives error message:

Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.

Then this is the fix:

sudo mkdir /sys/fs/cgroup/systemd sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

Run WSL + CUDA + Docker + Jupyter + TensorFlow

Finally, Jupyter notebook can be used in WSL to run popular AI framework code, like TensorFlow. Nvidia’s official tutorial runs TensorFlow with Jupyter notebook in Docker image tensorflow/tensorflow:latest-gpu-py3-jupyter. However, this docker image is no longer updated and gives errors. Now there is another up-to-date image tensorflow/tensorflow:latest-gpu-jupyter. In WSL, run :

docker run -it --gpus all -p 8889:8888 tensorflow/tensorflow:latest-gpu-jupyter

Here container’s port 8888 is mapped to WSL’s port 8889, because on Windows I am already running an instance of Jupyter notebook, which already occupies port 8888.

Dock shows the following messages:

________ _______________ ___ __/__________________________________ ____/__ /________ __ __ / _ _ \_ __ \_ ___/ __ \_ ___/_ /_ __ /_ __ \_ | /| / / _ / / __/ / / /(__ )/ /_/ / / _ __/ _ / / /_/ /_ |/ |/ / /_/ \___//_/ /_//____/ \____//_/ /_/ /_/ \____/____/|__/

WARNING: You are running this container as root, which can cause new files in mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...

[I 17:15:59.736 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret jupyter_http_over_ws extension initialized. Listening on /http_over_websocket [I 17:15:59.947 NotebookApp] Serving notebooks from local directory: /tf [I 17:15:59.948 NotebookApp] The Jupyter Notebook is running at: [I 17:15:59.948 NotebookApp] http://ccd3c60790ab:8888/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef [I 17:15:59.948 NotebookApp] or http://127.0.0.1:8888/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef [I 17:15:59.948 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 17:15:59.953 NotebookApp]

To access the notebook, open this file in a browser: file:///root/.local/share/jupyter/runtime/nbserver-1-open.html Or copy and paste one of these URLs: http://ccd3c60790ab:8888/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef or http://127.0.0.1:8888/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef

Now take the last URI, replace “127.0.0.1:8888” with “localhost:8889”, go to Windows and launch browser, paste the URI: http://localhost:8889/?token=8b1969bc9a278498fd5debe119feb6d86130850166425bef, The browser should load the Jupyter notebook with built-in TensorFlow tutorial:

However, running the cells in the notebook, it shows some errors. If it shows an error of git not found (“FileNotFoundError: No such file or directory: 'git'”), then run these commands in a cell to install git:

!apt update !apt install -y git

If the python code throws an error “InternalError: Blas GEMM launch failed” or “InternalError: Blas GEMV launch failed”, the fix is to run these code:

#import tensorflow as tf physical_devices = tf.config.list_physical_devices('GPU') tf.config.experimental.set_memory_growth(physical_devices[0], True)

The it should work:

Encoding and decoding video with GPU in WSL?

Not supported yet. To test GPU encoding and decoding, the latest version of FFmpeg can be installed:

sudo add-apt-repository ppa:savoury1/graphics sudo add-apt-repository ppa:savoury1/multimedia sudo add-apt-repository ppa:savoury1/ffmpeg4 sudo apt-get update sudo apt-get upgrade && sudo apt-get dist-upgrade sudo apt-get install ffmpeg

Then install the Nvidia encoding/decoding libraries:

sudo apt install libnvidia-decode-455 sudo apt install libnvidia-encode-455

Now try to use FFmpeg to decode a video and encode it with hevc_nvenc:

ffmpeg -hwaccel auto -i input.mkv -map 0:v:0 -map 0:a -map_metadata 0 -loglevel verbose -c:v hevc_nvenc -profile:v main10 -pix_fmt yuv420p10le -preset slow -tune film -b_adapt 2 -b_ref_mode middle -bf 3 -rc vbr_hq -deblock 0:0 -rc-lookahead 25 -rc_lookahead 25 -g 250 -keyint_min 23 -refs 4 -sc_threshold 40 -temporal_aq 1 -spatial_aq 1 -nonref_p 1 -c:a aac -ar 48000 -b:a 256k -ac 6 -b:v 2048k output.nvenc.mp4 –y

It does not work and gives the following error for decoding the input video:

[h264 @ 0x55c4517dd540] decoder->cvdl->cuvidGetDecoderCaps(&caps) failed -> CUDA_ERROR_INVALID_DEVICE: invalid device ordinal [h264 @ 0x55c4517dd540] Failed setup for format cuda: hwaccel initialisation returned error.

And it gives the following error for encoding the output video:

[hevc_nvenc @ 0x55c451065b00] OpenEncodeSessionEx failed: unsupported device (2): (no details) [hevc_nvenc @ 0x55c451065b00] Nvenc unloaded Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

I searched around and found a video of WSLConf’s CUDA session, where Nvidia conforms GPU encoding/decoding is not yet supported in WSL, and will come in the future:

Update code font from Consolas to Cascadia Code with ligature

Sun, 31 May 2020 00:00:00 GMT

A decade ago, I blogged that I switched my code font from Courier New to Consolas. They are both monospaced. The difference is:

Courier New is an old font introduced in Windows 3.1.
Consolas is introduced with Windows Vista/Office 2007/Visual Studio 2010. It always uses ClearType, which is designed for LCD screens and other flat panels which arrange their pixels in vertical stripes of red, green and blue.

After using Consolas for 10+ years, now I am switching code font to Cascadia Code.

Cascadia and Windows Terminal

A week ago, Widows Terminal 1.0 is released by Microsoft, along with a dedicated Cascadia fonts. Cascadia is open source and is named after Widows Terminal’s code name. I personally prefer the name “Seattle”, where I live.

Microsoft actually released 2 Cascadia fonts along with Windows Terminal:

Cascadia mono: the normal monospaced font
Cascadia code: the fancy version of Cascadia mono, with programming ligatures
Cascadia mono PL: Cascadia mono plus powerline symbols
Cascadia code PL: Cascadia code plus powerline symbols

They are designed by Aaron Bell from Saja Typeworks. Microsoft recommends Cascadia Code for terminal apps and code editors.

The font design looks good for me, except number 7 and letter f looks inconsistent with others:

Text ligature

Ligature means two or more graphemes are joined as a single glyph. The most example is that e and t can be joined as &. The following are examples of joined letters:

Here are some ligatures examples in Unicode (Multiple letters/single ligature/Unicode/HTML):

A E/Æ/U+00C6/Æ
a e/æ/U+00E6/æ
O E/Œ/U+0152/Œ
o e/œ/U+0153/œ
f f/ﬀ/U+FB00/ﬀ
f‌ i/ﬁ/U+FB01/ﬁ
i j/ĳ/U+0133/ĳ
s t/ﬆ/U+FB06/ﬆ
f f i/ﬃ/U+FB03/ﬃ

CSS also support ligature with properties font-feature-settings and font-variant-ligatures.

I used to practice calligraphy in Chinese and English, but personally I am not in favor of ligatures.

Code ligature

Programing ligature means to join code symbols as a single glyph. For example, in terminal or code editor,

“greater than or equal to” operator >= can be joined as ≥
“not equal” operator != cam ne joined as ≠
“double arrow”operator => can be joined as ⇒

Recently code ligature become popular because of the Fira Code font based on Fira Mono from Mozilla, Monoid font, Hasklig font, etc. Here is a detailed comparison.

Ligate in programming language is said to improve readability. Personally I do not have a strong preference. There are also some people who do not like ligature at all. In Cascadia Code, the following are some examples of how programming operator characters are joined as ligatures in this font:

++ -- ** && !! || %% ;; ?? +++ --- *** www # ## ### #### /= ^= =~ ~- -~ ~~ ~= ~@ ?. #{ #( #_ #_( #[ ]# #? #: #= #! <= >= <> <<< >>> == != =/= === !== =!= =:= <== ==> <=> => <<= =>> >>= =<< <=< >=> << >> -< >- <<- ->> >>- -<< <-- --> <- <-> -> <$ <$> $> <+ <+> +> <* <*> *> <~ <~> ~> </ /> </>  <||| <|| <| <|> |> ||> |||> .. ... :: ::: .- .= := ::= ..< <: :> :< >: __ -| _|_ |- |= ||= [| |] {| |} 0xFF 1x2

Powerline

Powerline was a status line plugin for vim. Powerline symbols (U+E0A0 to U+E0A2 and U+E0B0 to U+E0B3) are extra box-drawing characters that could be useful for displaying status.

U+E0A0
U+E0A1
U+E0A2
U+E0B0
U+E0B1
U+E0B2
U+E0B3

Install and configure

Cascadia Code is installed along with Windows Terminal. You can also download and install Cascadia Code PL from official release page: https://github.com/microsoft/cascadia-code/releases. With chocolatey, it can be installed by: choco install cascadiacodepl. You can also manually install it for Linux etc..

In Windows Terminal, the default font is Cascadia Mono.

Open settings. In the JSON configuration, under profiles/defaults, add "fontFace": "Cascadia Code PL", and save. The terminal instantly becomes:

To make usage of powerline symbols, you can install oh-my-posh for PowerShell, and install oh-my-zsh for WSL then switch the theme to “agnoster”:

In Visual Studio, to use Cascadia Code, In Tools/Options, search for font, change it to Cascadia:

In Visual Studio Code, open File/Preferences/Settings, search for font, and add it to the head:

Then search for ligature, turn it on ("editor.fontLigatures": true). On Windows, the ligatures will be shown in VS Code. On Ubuntu, you need to restart VS Code.

Installing SQL Server 2017/2019 LocalDB and resolve the engine versioning problem

Wed, 29 Apr 2020 00:00:00 GMT

SQL Server LocalDB is a minimal SQL Server database engine, it can be installed and used with zero configuration.

Get the installer and install

SQL Server LocalDB setup file is included in SQL Server Express. SQL Server 2019 can be downloaded from https://go.microsoft.com/fwlink/?LinkID=866658, and SQL Server 2017 can be downloaded from: https://go.microsoft.com/fwlink/?LinkID=853017.

After unzip the downloaded package, please run root\x64\Setup\x64\SQLLOCALDB.MSI to install.

When it is done, code can connect to it with a connection string, or SQL Server Management Studio can be used to manage the databases by connecting to (LocalDB)\MSSQLLocalDB:

Resolve the engine connectivity/versioning issue

After SQL Server LocalDB 2017/2019 is installed, connecting with SSNS may fail with an error:

In command line, trying to manage it with the sqllocaldb command may also fail with an error:

D:\ λ sqllocaldb info MSSQLLocalDB

D:\ λ sqllocaldb info MSSQLLocalDB Printing of LocalDB instance "MSSQLLocalDB" information failed because of the following error: Unexpected error occurred inside a LocalDB instance API method call. See the Windows Application event log for error details.

This is caused by engine versioning issue, which can be viewed in Windows Event Viewer:

LocalDB parent instance version is invalid: MSSQL13E.LOCALDB

Apparently “MSSQL13E” is incorrect. SQL Server 2016 is v13, SQL Server 2017 should be v14, and SQL Server 2019 should be v15. There are 2 ways to fix this:

The default instance can be deleted and recreated:

sqllocaldb stop mssqllocaldb sqllocaldb delete mssqllocaldb sqllocaldb create MSSQLLocalDB

Or the version info can be manually updated in Registry: Computer\HKEY_CURRENT_USER\Software\Microsoft\Microsoft SQL Server\UserInstances\{2DD3D445-34C1-4251-B67D-7DFEED432A87}

Just change ParentInstance to MSSQL14E.LOCALDB or MSSQL15E.LOCALDB.

Then SQL Server LocalDB can be managed by command line or SSMS:

D:\ λ sqllocaldb info MSSQLLocalDB Name: MSSQLLocalDB Version: 15.0.2000.5 Shared name: Owner: PC\dixin Auto-create: Yes State: Stopped Last start time: 4/28/2020 5:31:14 PM Instance pipe name:

Entity Framework Core and LINQ to Entities in Depth (8) Optimistic Concurrency

Tue, 15 Oct 2019 00:00:00 GMT

[LINQ via C# series]

[Entity Framework Core (EF Core) series]

[Entity Framework (EF) series]

Conflicts can occur if the same data is read and changed concurrently. Generally, there are 2 concurrency control approaches:

Pessimistic concurrency: one database client can lock the data being accessed, in order to prevent other database clients to change that same data concurrently.
Optimistic concurrency: Data is not locked in the database for client to CRUD. Any database client is allowed to read and change any data concurrently. As a result, concurrency conflicts can happen. This is how EF/Core work with database.

To demonstrate the behavior of EF/Core for concurrency, the following DbReaderWriter type is defined as database CRUD client:

internal partial class DbReaderWriter : IDisposable
{
    private readonly DbContext context;

    internal DbReaderWriter(DbContext context) => this.context = context;

    internal TEntity Read<TEntity>(params object[] keys) where TEntity : class => 
        this.context.Set<TEntity>().Find(keys);

    internal int Write(Action change)
    {
        change();
        return this.context.SaveChanges();
    }

    internal DbSet<TEntity> Set<TEntity>() where TEntity : class => this.context.Set<TEntity>();

    public void Dispose() => this.context.Dispose();
}

Multiple DbReaderWriter instances can be be used to read and write data concurrently. For example:

internal static partial class Concurrency
{
    internal static void NoCheck(
        DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3)
    {
        int id = 1;
        ProductCategory categoryCopy1 = readerWriter1.Read<ProductCategory>(id);
        ProductCategory categoryCopy2 = readerWriter2.Read<ProductCategory>(id);

        readerWriter1.Write(() => categoryCopy1.Name = nameof(readerWriter1));
        // exec sp_executesql N'SET NOCOUNT ON;
        // UPDATE [Production].[ProductCategory] SET [Name] = @p0
        // WHERE [ProductCategoryID] = @p1;
        // SELECT @@ROWCOUNT;
        // ',N'@p1 int,@p0 nvarchar(50)',@p1=1,@p0=N'readerWriter1'
        readerWriter2.Write(() => categoryCopy2.Name = nameof(readerWriter2)); // Last client wins.
        // exec sp_executesql N'SET NOCOUNT ON;
        // UPDATE [Production].[ProductCategory] SET [Name] = @p0
        // WHERE [ProductCategoryID] = @p1;
        // SELECT @@ROWCOUNT;
        // ',N'@p1 int,@p0 nvarchar(50)',@p1=1,@p0=N'readerWriter2'

        ProductCategory category3 = readerWriter3.Read<ProductCategory>(id);
        category3.Name.WriteLine(); // readerWriter2
    }
}

In this example, multiple DbReaderWriter instances read and write data concurrently:

readerWriter1 reads category “Bikes”
readerWriter2 reads category “Bikes”. These 2 entities are independent because they are are from different DbContext instances.
readerWriter1 updates category’s name from “Bikes” to “readerWriter1”. As previously discussed, by default EF/Core locate the category with its primary key.
In database, this category’s name is no longer “Bikes”
readerWriter2 updates category’s name from “Bikes” to “readerWriter2”. It locates the category with its primary key as well. The primary key is unchanged, so the same category can be located and the name can be changed.
So later when readerWriter3 reads the entity with the same primary key, the category entity’s Name is “readerWriter2”.

Detect Concurrency conflicts

Concurrency conflicts can be detected by checking entities’ property values besides primary keys. To required EF/Core to check a certain property, just add a System.ComponentModel.DataAnnotations.ConcurrencyCheckAttribute to it. Remember when defining ProductPhoto entity, its ModifiedDate has a [ConcurrencyCheck] attribute:

public partial class ProductPhoto
{
    [ConcurrencyCheck]
    public DateTime ModifiedDate { get; set; }
}

This property is also called the concurrency token. When EF/Core translate changes of a photo, ModifiedDate property is checked along with the primary key to locate the photo:

internal static void ConcurrencyCheck(DbReaderWriter readerWriter1, DbReaderWriter readerWriter2)
{
    int id = 1;
    ProductPhoto photoCopy1 = readerWriter1.Read<ProductPhoto>(id);
    ProductPhoto photoCopy2 = readerWriter2.Read<ProductPhoto>(id);

    readerWriter1.Write(() =>
    {
        photoCopy1.LargePhotoFileName = nameof(readerWriter1);
        photoCopy1.ModifiedDate = DateTime.Now;
    });
    // exec sp_executesql N'SET NOCOUNT ON;
    // UPDATE [Production].[ProductPhoto] SET [LargePhotoFileName] = @p0, [ModifiedDate] = @p1
    // WHERE [ProductPhotoID] = @p2 AND [ModifiedDate] = @p3;
    // SELECT @@ROWCOUNT;
    // ',N'@p2 int,@p0 nvarchar(50),@p1 datetime2(7),@p3 datetime2(7)',@p2=1,@p0=N'readerWriter1',@p1='2017-01-25 22:04:25.9292433',@p3='2008-04-30 00:00:00'
    readerWriter2.Write(() =>
    {
        photoCopy2.LargePhotoFileName = nameof(readerWriter2);
        photoCopy2.ModifiedDate = DateTime.Now;
    });
    // exec sp_executesql N'SET NOCOUNT ON;
    // UPDATE [Production].[ProductPhoto] SET [LargePhotoFileName] = @p0, [ModifiedDate] = @p1
    // WHERE [ProductPhotoID] = @p2 AND [ModifiedDate] = @p3;
    // SELECT @@ROWCOUNT;
    // ',N'@p2 int,@p0 nvarchar(50),@p1 datetime2(7),@p3 datetime2(7)',@p2=1,@p0=N'readerWriter2',@p1='2017-01-25 22:04:59.1792263',@p3='2008-04-30 00:00:00'
}

In the translated SQL statement, the WHERE clause contains primary key and the original concurrency token. The following is how EF/Core check the concurrency conflicts:

readerWriter1 reads photo with primary key 1, and modified date “2008-04-30 00:00:00”
readerWriter2 reads the same photo with primary key 1, and modified date “2008-04-30 00:00:00”
readerWriter1 locates the photo with primary key and original modified date, and update its large photo file name and modified date.
In database the photo’s modified date is no longer the original value “2008-04-30 00:00:00”
readerWriter2 tries to locate the photo with primary key and original modified date. However the provided modified date is outdated. EF/Core detect that 0 row is updated by the translated SQL, and throws DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded. See http://go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.

Another option for concurrency check is System.ComponentModel.DataAnnotations.TimestampAttribute. It can only be used for a byte[] property, which is mapped from a rowversion (timestamp) column. For SQL database, these 2 terms, rowversion and timestamp, are the same thing. timestamp is just a synonym of rowversion data type. A row’s non-nullable rowversion column is a 8 bytes (binary(8)) counter maintained by database, its value increases for each change of the row.

Microsoft’s AdventureWorks sample database does not have such a rowversion column, so create one for the Production.Product table:

ALTER TABLE [Production].[Product] ADD [RowVersion] rowversion NOT NULL
GO

Then define the mapping property for Product entity:

public partial class Product
{
    [DatabaseGenerated(DatabaseGeneratedOption.Computed)]
    [Timestamp]
    public byte[] RowVersion { get; set; }

    [NotMapped]
    public string RowVersionString =>
        $"0x{BitConverter.ToUInt64(this.RowVersion.Reverse().ToArray(), 0).ToString("X16")}";
}

Now RowVersion property is the concurrency token. Regarding database automatically increases the RowVersion value, Rowversion also has the [DatabaseGenerated(DatabaseGeneratedOption.Computed)] attribute. The other RowVersionString property returns a readable representation of the byte array returned by RowVersion. It is not a part of the object-relational mapping, so it has a [NotMapped] attribute. The following example updates and and deletes the same product concurrently:

internal static void RowVersion(DbReaderWriter readerWriter1, DbReaderWriter readerWriter2)
{
    int id = 995;
    Product productCopy1 = readerWriter1.Read<Product>(id);
    productCopy1.RowVersionString.WriteLine(); // 0x0000000000000803

    Product productCopy2 = readerWriter2.Read<Product>(id);
    productCopy2.RowVersionString.WriteLine(); // 0x0000000000000803

    readerWriter1.Write(() => productCopy1.Name = nameof(readerWriter1));
    // exec sp_executesql N'SET NOCOUNT ON;
    // UPDATE [Production].[Product] SET [Name] = @p0
    // WHERE [ProductID] = @p1 AND [RowVersion] = @p2;
    // SELECT [RowVersion]
    // FROM [Production].[Product]
    // WHERE @@ROWCOUNT = 1 AND [ProductID] = @p1;
    // ',N'@p1 int,@p0 nvarchar(50),@p2 varbinary(8)',@p1=995,@p0=N'readerWriter1',@p2=0x0000000000000803
    productCopy1.RowVersionString.WriteLine(); // 0x00000000000324B1
    readerWriter2.Write(() => readerWriter2.Set<Product>().Remove(productCopy2));
    // exec sp_executesql N'SET NOCOUNT ON;
    // DELETE FROM [Production].[Product]
    // WHERE [ProductID] = @p0 AND [RowVersion] = @p1;
    // SELECT @@ROWCOUNT;
    // ',N'@p0 int,@p1 varbinary(8)',@p0=995,@p1=0x0000000000000803
}

When updating and deleting photo entities, its auto generated RowVersion property value is checked too. So this is how it works:

readerWriter1 reads product with primary key 995 and row version 0x0000000000000803
readerWriter2 reads product with the same primary key 995 and row version 0x0000000000000803
readerWriter1 locates the photo with primary key and original row version, and update its name. Database automatically increases the photo’s row version. Since the row version is specified as [DatabaseGenerated(DatabaseGeneratedOption.Computed)], EF/Core also locate the photo with the primary key to query the increased row version, and update the entity at client side.
In database the product’s row version is no longer 0x0000000000000803.
Then readerWriter2 tries to locate the product with primary key and original row version, and delete it. No product can be found with outdated row version, EF/Core detect that 0 row is deleted, and throws DbUpdateConcurrencyException.

Resolve concurrency conflicts

DbUpdateConcurrencyException is thrown when SaveChanges detects concurrency conflict:

namespace Microsoft.EntityFrameworkCore
{
    public class DbUpdateException : Exception
    {
        public virtual IReadOnlyList<EntityEntry> Entries { get; }

        // Other members.
    }

    public class DbUpdateConcurrencyException : DbUpdateException
    {
        // Members.
    }
}

Inherited from DbUpdateException, DbUpdateConcurrencyException has an Entries property. Entries returns a sequence of EntityEntry instances, representing the conflicting entities’ tracking information. The basic idea of resolving concurrency conflicts, is to handle DbUpdateConcurrencyException and retry SaveChanges:

internal partial class DbReaderWriter
{
    internal int Write(Action change, Action<DbUpdateConcurrencyException> handleException, int retryCount = 3)
    {
        change();
        for (int retry = 1; retry < retryCount; retry++)
        {
            try
            {
                return this.context.SaveChanges();
            }
            catch (DbUpdateConcurrencyException exception)
            {
                handleException(exception);
            }
        }
        return this.context.SaveChanges();
    }
}

In the above Write overload, if SaveChanges throws DbUpdateConcurrencyException, the handleException function is called. This function is expected to handle the exception and resolve the conflicts properly. Then SaveChanges is called again. If the last retry of SaveChanges still throws DbUpdateConcurrencyException, the exception is thrown to the caller.

Retain database values (database wins)

Similar to previous examples, the following example has multiple DbReaderWriter instances to update a product concurrently:

internal static void UpdateProduct(
    DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3,
    Action<EntityEntry> resolveConflicts)
{
    int id = 950;
    Product productCopy1 = readerWriter1.Read<Product>(id);
    Product productCopy2 = readerWriter2.Read<Product>(id);

    readerWriter1.Write(() =>
    {
        productCopy1.Name = nameof(readerWriter1);
        productCopy1.ListPrice = 100.0000M;
    });
    readerWriter2.Write(
        change: () =>
        {
            productCopy2.Name = nameof(readerWriter2);
            productCopy2.ProductSubcategoryID = 1;
        },
        handleException: exception =>
        {
            EntityEntry tracking = exception.Entries.Single();
            Product original = (Product)tracking.OriginalValues.ToObject();
            Product current = (Product)tracking.CurrentValues.ToObject();
            Product database = productCopy1; // Values saved in database.
            $"Original:  ({original.Name},   {original.ListPrice}, {original.ProductSubcategoryID}, {original.RowVersionString})"
                        .WriteLine();
            $"Database:  ({database.Name}, {database.ListPrice}, {database.ProductSubcategoryID}, {database.RowVersionString})"
                .WriteLine();
            $"Update to: ({current.Name}, {current.ListPrice}, {current.ProductSubcategoryID})"
                .WriteLine();

            resolveConflicts(tracking);
        });

    Product resolved = readerWriter3.Read<Product>(id);
    $"Resolved:  ({resolved.Name}, {resolved.ListPrice}, {resolved.ProductSubcategoryID}, {resolved.RowVersionString})"
        .WriteLine();
}

This is how it works with concurrency conflicts:

readerWriter1 reads product with primary key 950, and RowVersion 0x00000000000007D1
readerWriter2 reads product with the same primary key 950, and RowVersion 0x00000000000007D1
readerWriter1 locates product with primary key and original RowVersion 0x00000000000007D1, and updates product’s name and list price. Database automatically increases the product’s row version
In database the product’s row version is no longer 0x00000000000007D1.
readerWriter2 tries to locate product with primary key and original RowVersion, and update product’s name and subcategory.
readerWriter2 fails to update product, because it cannot locate the product with original RowVersion 0x00000000000007D1. Again, no product can be found with outdated row version, DbUpdateConcurrencyException is thrown.

As a result, the handleException function specified for readWriter2 is called, it retrieves the conflicting product’s tracking information from DbUpdateConcurrencyException.Entries, and logs these information:

product’s original property values read by readerWriter2 before the changes
product’s property values in database at this moment, which are already updated readerWriter1
product’s current property values after changes, which readerWriter2 fails to save to database.

Then handleException calls resolveConflicts function to actually resolve the conflict. Then readerWriter2 retries to save the product changes again. This time, SaveChanges should succeed, because there is no conflicts anymore (In this example, there are only 2 database clients reading/writing data concurrently. In reality, the concurrency can be higher, an appropriate retry count or retry strategy should be specified.). Eventually, readerWriter3 reads the product from database, verify its property values.

There are several options to implement the resolveConflicts function to resolves the conflicts. One simple option, called “database wins”, is to simply give up the client update, and let database retain whatever values it has for that entity. This seems to be easy to just catch DbUpdateConcurrencyException and do nothing, then database naturally wins, and retains its values:

internal partial class DbReaderWriter
{
    internal int WriteDatabaseWins(Action change)
    {
        change();
        try
        {
            return this.context.SaveChanges();
        }
        catch (DbUpdateConcurrencyException)
        {
            return 0; // this.context is in a corrupted state.
        }
    }
}

However, this way leaves the DbContext, the conflicting entity, and the entity’s tracking information in a corrupted state. For the caller, since the change saving is done, the entity’s property values should be in sync with database values, but the values are actually out of sync and still conflicting. Also, the entity has a tracking state Modified after change saving is done. So the safe approach is to reload and refresh the entity’s values and tracking information:

internal static void DatabaseWins(
    DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3)
{
    UpdateProduct(readerWriter1, readerWriter2, readerWriter3, resolveConflicts: tracking =>
    {
        tracking.State.WriteLine(); // Modified
        tracking.Property(nameof(Product.Name)).IsModified.WriteLine(); // True
        tracking.Property(nameof(Product.ListPrice)).IsModified.WriteLine(); // False
        tracking.Property(nameof(Product.ProductSubcategoryID)).IsModified.WriteLine(); // True

        tracking.Reload(); // Execute query.

        tracking.State.WriteLine(); // Unchanged
        tracking.Property(nameof(Product.Name)).IsModified.WriteLine(); // False
        tracking.Property(nameof(Product.ListPrice)).IsModified.WriteLine(); // False
        tracking.Property(nameof(Product.ProductSubcategoryID)).IsModified.WriteLine(); // False
    });
    // Original:  (ML Crankset,   256.4900, 8, 0x00000000000007D1)
    // Database:  (readerWriter1, 100.0000, 8, 0x0000000000036335)
    // Update to: (readerWriter2, 256.4900, 1)
    // Resolved:  (readerWriter1, 100.0000, 8, 0x0000000000036335)
}

UpdateProduct is called with a resolveConflicts function, which resolves the conflict by calling Reload method on the EntityEntry instance representing the conflicting product’s tracking information:

EntityEntry.Reload executes a SELECT statement to read the product’s property values from database, then refresh the product entity and all tracking information. The product’s property values, the tracked original property values before changes, the tracked current property values after changes, are all refreshed to the queried database values. The entity tracking state is also refreshed to Unchanged.
At this moment, product has the same tracked original values and current values, as if it is just initially read from database, without changes.
When DbReaderWriter.Write’s retry logic calls SaveChanges again, no changed entity is detected. SaveChanges succeeds without executing any SQL, and returns 0. As expected, readerWriter2 does not update any value to database, and all values in database are retained.

Later, when readerWriter3 reads the product again, product has all values updated by readerWrtier1.

Overwrite database values (client wins)

Another simple option, called “client wins”, is to disregard values in database, and overwrite them with whatever data submitted from client.

internal static void ClientWins(
    DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3)
{
    UpdateProduct(readerWriter1, readerWriter2, readerWriter3, resolveConflicts: tracking =>
    {
        PropertyValues databaseValues = tracking.GetDatabaseValues();
        // Refresh original values, which go to WHERE clause of UPDATE statement.
        tracking.OriginalValues.SetValues(databaseValues);

        tracking.State.WriteLine(); // Modified
        tracking.Property(nameof(Product.Name)).IsModified.WriteLine(); // True
        tracking.Property(nameof(Product.ListPrice)).IsModified.WriteLine(); // True
        tracking.Property(nameof(Product.ProductSubcategoryID)).IsModified.WriteLine(); // True
    });
    // Original:  (ML Crankset,   256.4900, 8, 0x00000000000007D1)
    // Database:  (readerWriter1, 100.0000, 8, 0x0000000000036336)
    // Update to: (readerWriter2, 256.4900, 1)
    // Resolved:  (readerWriter2, 256.4900, 1, 0x0000000000036337)
}

The same conflict is resolved differently:

EntityEntry.GetDatabaseValues executes a SELECT statement to read the product’s property values from database, including the updated row version. This call does not impact the product values or tracking information.
Manually set the tracked original property values to the queried database values. The entity tracking state is still Changed. The original property values become all different from tracked current property values. So all product properties are tracked as modified.
At this moment, the product has tracked original values updated, and keeps all tracked current values, as if it is read from database after readerWriter1 updates the name and list price, and then have all properties values changed.
When DbReaderWriter.Write’s retry logic calls SaveChanges again, product changes are detected to submit. So EF/Core translate the product change to a UPDATE statement. In the SET clause, since there are 3 properties tracked as modified, 3 columns are set. In the WHERE clause, to locate the product, the tracked original row version has been set to the updated value from database. This time product can be located, and all 3 properties are updated. SaveChanges succeeds and returns 1. As expected, readerWriter2 updates all value to database.

Later, when readerWriter3 reads the product again, product has all values updated by readerWrter2.

Merge with database values

A more complex but useful option, is to merge the client values and database values. For each property:

If original value is different from database value, which means database value is already updated by other concurrent client, then give up updating this property, and retain the database value
If original value is the same as database value, which means no concurrency conflict for this property, then process normally to submit the change

internal static void MergeClientAndDatabase(
    DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3)
{
    UpdateProduct(readerWriter1, readerWriter2, readerWriter3, resolveConflicts: tracking =>
    {
        PropertyValues databaseValues = tracking.GetDatabaseValues(); // Execute query.
        PropertyValues originalValues = tracking.OriginalValues.Clone();
        // Refresh original values, which go to WHERE clause.
        tracking.OriginalValues.SetValues(databaseValues);
        // If database has an different value for a property, then retain the database value.
#if EF
        databaseValues.PropertyNames // Navigation properties are not included.
            .Where(property => !object.Equals(originalValues[property], databaseValues[property]))
            .ForEach(property => tracking.Property(property).IsModified = false);
#else
        databaseValues.Properties // Navigation properties are not included.
            .Where(property => !object.Equals(originalValues[property.Name], databaseValues[property.Name]))
            .ForEach(property => tracking.Property(property.Name).IsModified = false);
#endif
        tracking.State.WriteLine(); // Modified
        tracking.Property(nameof(Product.Name)).IsModified.WriteLine(); // False
        tracking.Property(nameof(Product.ListPrice)).IsModified.WriteLine(); // False
        tracking.Property(nameof(Product.ProductSubcategoryID)).IsModified.WriteLine(); // True
    });
    // Original:  (ML Crankset,   256.4900, 8, 0x00000000000007D1)
    // Database:  (readerWriter1, 100.0000, 8, 0x0000000000036338)
    // Update to: (readerWriter2, 256.4900, 1)
    // Resolved:  (readerWriter1, 100.0000, 1, 0x0000000000036339)
}

With this approach:

Again, EntityEntry.GetDatabaseValues executes a SELECT statement to read the product’s property values from database, including the updated row version.
Backup tracked original values, then refresh conflict.OriginalValues to the database values, so that these values can go to the translated WHERE clause. Again, the entity tracking state is still Changed. The original property values become all different from tracked current property values. So all product values are tracked as modified and should go to SET clause.
For each property, if the backed original value is different from the database value, it means this property is changed by other client and there is concurrency conflict. In this case, revert this property’s tracking status to unmodified. The name and list price are reverted.
At this moment, the product has tracked original values updated, and only keeps tracked current value of subcategory, as if it is read from database after readerWriter1 updates the name and list price, and then only have subcategory changed, which has no conflict.
When DbReaderWriter.Write’s retry logic calls SaveChanges again, product changes are detected to submit. Here only subcategory is updated to database. SaveChanges succeeds and returns 1. As expected, readerWriter2 only updates value without conflict, the other conflicted values are retained.

Later, when readerWriter3 reads the product, product has name and list price values updated by readerWrtier1, and has subcategory updated by readerWriter2.

Save changes with concurrency conflict handling

Similar to above DbReaderWriter.Write method, a general SaveChanges extension method for DbContext can be defined to handle concurrency conflicts and apply simple retry logic:

public static partial class DbContextExtensions
{
    public static int SaveChanges(
        this DbContext context, Action<IEnumerable<EntityEntry>> resolveConflicts, int retryCount = 3)
    {
        if (retryCount <= 0)
        {
            throw new ArgumentOutOfRangeException(nameof(retryCount));
        }

        for (int retry = 1; retry < retryCount; retry++)
        {
            try
            {
                return context.SaveChanges();
            }
            catch (DbUpdateConcurrencyException exception) when (retry < retryCount)
            {
                resolveConflicts(exception.Entries);
            }
        }
        return context.SaveChanges();
    }
}

To apply custom retry logic, Microsoft provides EnterpriseLibrary.TransientFaultHandling NuGet package (Exception Handling Application Block) for .NET Framework. It has been ported to .NET Core for this tutorial, as EnterpriseLibrary.TransientFaultHandling.Core NuGet package. can be used. With this library, a SaveChanges overload with customizable retry logic can be easily defined:

public class TransientDetection<TException> : ITransientErrorDetectionStrategy
    where TException : Exception
{
    public bool IsTransient(Exception ex) => ex is TException;
}

public static partial class DbContextExtensions
{
    public static int SaveChanges(
        this DbContext context, Action<IEnumerable<EntityEntry>> resolveConflicts, RetryStrategy retryStrategy)
    {
        RetryPolicy retryPolicy = new RetryPolicy(
            errorDetectionStrategy: new TransientDetection<DbUpdateConcurrencyException>(),
            retryStrategy: retryStrategy);
        retryPolicy.Retrying += (sender, e) =>
            resolveConflicts(((DbUpdateConcurrencyException)e.LastException).Entries);
        return retryPolicy.ExecuteAction(context.SaveChanges);
    }
}

Here Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.ITransientErrorDetectionStrategy is the contract to detect each exception, and determine whether the exception is transient and the operation should be retried. Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.RetryStrategy is the contract of retry logic. Then Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling.RetryPolicy executes the operation with the specified exception detection, exception handling, and retry logic.

As discussed above, to resolve a concurrency conflict, the entity and its tracking information need to be refreshed. So the more specific SaveChanges overloads can be implemented by applying refresh for each conflict:

public enum RefreshConflict
{
    StoreWins,

    ClientWins,

    MergeClientAndStore
}

public static partial class DbContextExtensions
{
    public static int SaveChanges(this DbContext context, RefreshConflict refreshMode, int retryCount = 3)
    {
        if (retryCount <= 0)
        {
            throw new ArgumentOutOfRangeException(nameof(retryCount));
        }

        return context.SaveChanges(
            conflicts => conflicts.ForEach(tracking => tracking.Refresh(refreshMode)), retryCount);
    }

    public static int SaveChanges(
        this DbContext context, RefreshConflict refreshMode, RetryStrategy retryStrategy) =>
            context.SaveChanges(
                conflicts => conflicts.ForEach(tracking => tracking.Refresh(refreshMode)), retryStrategy);
}

A RefreshConflict enumeration has to be defined with 3 members to represent the 3 options discussed above: database wins, client wind, merge client and database.. And here the Refresh method is an extension method for EntityEntry:

public static EntityEntry Refresh(this EntityEntry tracking, RefreshConflict refreshMode)
{
    switch (refreshMode)
    {
        case RefreshConflict.StoreWins:
        {
            // When entity is already deleted in database, Reload sets tracking state to Detached.
            // When entity is already updated in database, Reload sets tracking state to Unchanged.
            tracking.Reload(); // Execute SELECT.
            // Hereafter, SaveChanges ignores this entity.
            break;
        }
        case RefreshConflict.ClientWins:
        {
            PropertyValues databaseValues = tracking.GetDatabaseValues(); // Execute SELECT.
            if (databaseValues == null)
            {
                // When entity is already deleted in database, there is nothing for client to win against.
                // Manually set tracking state to Detached.
                tracking.State = EntityState.Detached;
                // Hereafter, SaveChanges ignores this entity.
            }
            else
            {
                // When entity is already updated in database, refresh original values, which go to in WHERE clause.
                tracking.OriginalValues.SetValues(databaseValues);
                // Hereafter, SaveChanges executes UPDATE/DELETE for this entity, with refreshed values in WHERE clause.
            }
            break;
        }
        case RefreshConflict.MergeClientAndStore:
        {
            PropertyValues databaseValues = tracking.GetDatabaseValues(); // Execute SELECT.
            if (databaseValues == null)
            {
                // When entity is already deleted in database, there is nothing for client to merge with.
                // Manually set tracking state to Detached.
                tracking.State = EntityState.Detached;
                // Hereafter, SaveChanges ignores this entity.
            }
            else
            {
                // When entity is already updated, refresh original values, which go to WHERE clause.
                PropertyValues originalValues = tracking.OriginalValues.Clone();
                tracking.OriginalValues.SetValues(databaseValues);
                // If database has an different value for a property, then retain the database value.
#if EF
                databaseValues.PropertyNames // Navigation properties are not included.
                    .Where(property => !object.Equals(originalValues[property], databaseValues[property]))
                    .ForEach(property => tracking.Property(property).IsModified = false);
#else
                databaseValues.Properties // Navigation properties are not included.
                    .Where(property => !object.Equals(originalValues[property.Name], databaseValues[property.Name]))
                    .ForEach(property => tracking.Property(property.Name).IsModified = false);
#endif
                // Hereafter, SaveChanges executes UPDATE/DELETE for this entity, with refreshed values in WHERE clause.
            }
            break;
        }
    }
    return tracking;
}

EF already provides a System.Data.Entity.Core.Objects.RefreshMode enumeration, but it only has 2 members: StoreWins and ClientWins.

This Refresh extension method covers the update conflicts discussed above, as well as deletion conflicts. Now the these SaveChanges extension methods can be used to manage concurrency conflicts easily. For example:

internal static void SaveChanges(AdventureWorks adventureWorks1, AdventureWorks adventureWorks2)
{
    int id = 950;
    Product productCopy1 = adventureWorks1.Products.Find(id);
    Product productCopy2 = adventureWorks2.Products.Find(id);

    productCopy1.Name = nameof(adventureWorks1);
    productCopy1.ListPrice = 100;
    adventureWorks1.SaveChanges();

    productCopy2.Name = nameof(adventureWorks2);
    productCopy2.ProductSubcategoryID = 1;
    adventureWorks2.SaveChanges(RefreshConflict.MergeClientAndStore);
}

Entity Framework Core and LINQ to Entities in Depth (7) Data Changes and Transactions

Mon, 14 Oct 2019 00:00:00 GMT

[LINQ via C# series]

[Entity Framework Core (EF Core) series]

[Entity Framework (EF) series]

Besides LINQ to Entities queries, EF Core also provides rich APIs for data changes, with imperative paradigm.

Repository pattern and unit of work pattern

In EF Core, DbSet<T> implements repository pattern. Repositories can centralize data access for applications, and connect between the data source and the business logic. A DbSet<T> instance can be mapped to a database table, which is a repository for data CRUD (create, read, update and delete):

namespace Microsoft.EntityFrameworkCore

public abstract class DbSet<TEntity> : IQueryable<TEntity> // Other interfaces.

where TEntity : class

public virtual TEntity Find(params object[] keyValues);

public virtual EntityEntry<TEntity> Add(TEntity entity);

public virtual void AddRange(IEnumerable<TEntity> entities);

public virtual EntityEntry<TEntity> Remove(TEntity entity);

public virtual void RemoveRange(IEnumerable<TEntity> entities);

// Other members.

}

DbSet<T> implements IQueryable<T>, so that DbSet<T> can represent the data source to read from. DbSet<T>.Find is also provided to read entity by the primary keys. After reading, the retrieved data can be changed. Add and AddRange methods track the specified entities as to be created in the repository. Remove and RemoveRange methods track the specified entities as to be deleted in the repository.

As fore mentioned, a unit of work is a collection of data operations that should together or fail together as a unit. DbContext implements unit of work pattern:

namespace Microsoft.EntityFrameworkCore

public class DbContext : IDisposable, IInfrastructure<IServiceProvider>

public virtual DbSet<TEntity> Set<TEntity>() where TEntity : class;

public virtual ChangeTracker ChangeTracker { get; }

public virtual int SaveChanges();

public virtual void Dispose();

}

As the mapping of database, DbContext’s Set method returns the specified entity’s repositories. For example, calling AdventureWorks.Products is equivalent to calling AdventureWorks.Set<Product>. The entities tracking is done at the DbContext level, by its ChangeTracker. When DbContext.Submit is called, the tracked changes are submitted to database. When a unit of work is done, DbContext should be disposed.

Track entities and changes

DbContext.ChangeTracker property returns Microsoft.EntityFrameworkCore.ChangeTracking.ChangeTracker, which can track entities for the source DbContext:

namespace Microsoft.EntityFrameworkCore.ChangeTracking

public class ChangeTracker : IInfrastructure<IStateManager>

public virtual IEnumerable<EntityEntry> Entries();

public virtual IEnumerable<EntityEntry<TEntity>> Entries<TEntity>() where TEntity : class;

public virtual void DetectChanges();

public virtual bool HasChanges();

// Other members.

}

Each entity’s loading and tracking information is represented by Microsoft.EntityFrameworkCore.ChangeTracking.EntityEntry or Microsoft.EntityFrameworkCore.ChangeTracking.EntityEntry<TEntity>. The following is the non generic EntityEntry:

namespace Microsoft.EntityFrameworkCore.ChangeTracking

public class EntityEntry : IInfrastructure<InternalEntityEntry>

public virtual EntityState State { get; set; }

public virtual object Entity { get; }

public virtual PropertyEntry Property(string propertyName);

public virtual PropertyValues CurrentValues { get; }

public virtual PropertyValues OriginalValues { get; }

public virtual PropertyValues GetDatabaseValues();

public virtual void Reload();

// Other members.

}

Besides the loading information APIs discussed in previous part, EntityEntry also provides rich APIs for entity’s tracking information and state management:

· State returns the entity’s tracking state: Detached, Unchanged, Added, Deleted, or Modified.

· Entity property returns the tracked entity

· Property returns the specified property’s tracking information.

· CurrentValues returns the tracked entity’s current property values.

· OriginalValues returns the tracked entity’s original property values

· GetDatabaseValues instantly execute a SQL query to read entity’s property values from database, without updating current entity’s property values and tracking information.

· Reload also executes a SQL query to read the database values, and also update current entity’s property values, and all tracking information

The generic EntityEntry<TEntity> is just stronger typing:

namespace Microsoft.EntityFrameworkCore.ChangeTracking

public class EntityEntry<TEntity> : EntityEntry where TEntity : class

public virtual TEntity Entity { get; }

// Other members.

}

As fore mentioned in data loading part, DbContext.Entry also accepts an entity and return its EntityEntry<TEntity>/EntityEntry.

Track entities

By default, all entities read from repositories are tracked by the source DbContext. For example:

internal static void EntitiesFromSameDbContext(AdventureWorks adventureWorks)

Product productById = adventureWorks.Products

.Single(product => product.ProductID == 999);

adventureWorks.ChangeTracker.Entries().Count().WriteLine(); // 1

Product productByName = adventureWorks.Products

.Single(product => product.Name == "Road-750 Black, 52");

adventureWorks.ChangeTracker.Entries().Count().WriteLine(); // 1

object.ReferenceEquals(productById, productByName).WriteLine(); // True

}

The single result from the first LINQ to Entities query is tracked by DbContext. Later, the second query has a single result too. EF Core identifies both results map to the same data row of the same table, so they are reference to the same entity instance.

If data from repositories are not entities mapping to table rows, they cannot be tracked:

internal static void ObjectsFromSameDbContext(AdventureWorks adventureWorks)

var productById = adventureWorks.Products

.Select(product => new { ProductID = product.ProductID, Name = product.Name })

.Single(product => product.ProductID == 999);

var productByName = adventureWorks.Products

.Select(product => new { ProductID = product.ProductID, Name = product.Name })

.Single(product => product.Name == "Road-750 Black, 52");

adventureWorks.ChangeTracker.Entries().Count().WriteLine(); // 0

object.ReferenceEquals(productById, productByName).WriteLine(); // False

}

Here data is queries from repositories, and anonymous type instances are constructed on the fly. EF Core cannot decide if 2 arbitrary instances semantically represent the same piece of data in remote database. This time 2 query results are independent from each other.

Since the tracking is at DbContext scope. Entities of different DbContext instances belong to different units of work, and do not interfere each other:

internal static void EntitiesFromMultipleDbContexts()

Product productById;

Product productByName;

using (AdventureWorks adventureWorks = new AdventureWorks())

productById = adventureWorks.Products.Single(product => product.ProductID == 999);

using (AdventureWorks adventureWorks = new AdventureWorks())

productByName = adventureWorks.Products.Single(product => product.Name == "Road-750 Black, 52");

object.ReferenceEquals(productById, productByName).WriteLine(); // False.

}

Track entity changes and property changes

The following example demonstrate CRUD operations in the product repository, then examine all the tracking information:

internal static void EntityChanges(AdventureWorks adventureWorks)

Product create = new Product() { Name = nameof(create), ListPrice = 1 };

adventureWorks.Products.Add(create); // Create locally.

Product read = adventureWorks.Products.Single(product => product.ProductID == 999); // Read from remote to local.

IQueryable<Product> update = adventureWorks.Products

.Where(product => product.Name.Contains("HL"));

update.ForEach(product => product.ListPrice += 100); // Update locally.

IQueryable<Product> delete = adventureWorks.Products

.Where(product => product.Name.Contains("ML"));

adventureWorks.Products.RemoveRange(delete); // Delete locally.

adventureWorks.ChangeTracker.HasChanges().WriteLine(); // True

adventureWorks.ChangeTracker.Entries<Product>().ForEach(tracking =>

Product changed = tracking.Entity;

switch (tracking.State)

case EntityState.Added:

case EntityState.Deleted:

case EntityState.Unchanged:

$"{tracking.State}: {(changed.ProductID, changed.Name, changed.ListPrice)}".WriteLine();

break;

case EntityState.Modified:

Product original = (Product)tracking.OriginalValues.ToObject();

$"{tracking.State}: {(original.ProductID, original.Name, original.ListPrice)} => {(changed.ProductID, changed.Name, changed.ListPrice)}"

.WriteLine();

break;

});

// Added: (-2147482647, toCreate, 1)

// Unchanged: (999, Road-750 Black, 52, 539.9900)

// Modified: (951, HL Crankset, 404.9900) => (951, HL Crankset, 504.9900)

// Modified: (996, HL Bottom Bracket, 121.4900) => (996, HL Bottom Bracket, 221.4900)

// Deleted: (950, ML Crankset, 256.4900)

// Deleted: (995, ML Bottom Bracket, 101.2400)

}

If an entity is not read from a DbContext instance’s repositories, then it has nothing to do with that unit of work, and apparently is not tracked by that DbContext instance. DbSet<T> provides an Attach method to place an entity to the repository, and the DbContext tracks the entity as the Unchanged state:

internal static void Attach(AdventureWorks adventureWorks)

Product product = new Product() { ProductID = 950, Name = "ML Crankset", ListPrice = 539.99M };

adventureWorks.ChangeTracker.Entries().Count().WriteLine(); // 0

adventureWorks.Products.Attach(product);

adventureWorks.ChangeTracker.Entries().Count().WriteLine(); // 1

adventureWorks.ChangeTracker.Entries<Product>().Single().State.WriteLine(); // Unchanged

product.Name = "After attaching";

adventureWorks.ChangeTracker.Entries<Product>().Single().State.WriteLine(); // Modified

adventureWorks.ChangeTracker.Entries<Product>().WriteLines(tracking =>

$"{tracking.State}: {tracking.OriginalValues[nameof(Product.Name)]} => {tracking.CurrentValues[nameof(Product.Name)]}");

// Modified: ML Crankset => After attaching

}

Track relationship changes

The relationship of entities is also tracked. Remember Product’s foreign key ProductSubcategoryID is nullable. The following example reads a subcategory and its products, then delete the relationship. As a result, each navigation property is cleared to empty collection or null. And each related subcategory’s foreign key property value is synced to null, which is tracked:

internal static void RelationshipChanges(AdventureWorks adventureWorks)

ProductSubcategory subcategory = adventureWorks.ProductSubcategories

.Include(entity => entity.Products).Single(entity => entity.ProductSubcategoryID == 8);

subcategory.Products.Count.WriteLine(); // 2

subcategory.Products

.All(product => product.ProductSubcategory == subcategory).WriteLine(); // True

subcategory.Products.Clear();

// Equivalent to: subcategory.Products.ForEach(product => product.ProductSubcategory = null);

subcategory.Products.Count.WriteLine(); // 0

subcategory.Products

.All(product => product.ProductSubcategory == null).WriteLine(); // True

adventureWorks.ChangeTracker.Entries<Product>().ForEach(tracking =>

Product original = (Product)tracking.OriginalValues.ToObject();

Product changed = tracking.Entity;

$"{tracking.State}: {(original.ProductID, original.Name, original.ProductSubcategoryID)} => {(changed.ProductID, changed.Name, changed.ProductSubcategoryID)}".WriteLine();

});

// Modified: (950, ML Crankset, 8) => (950, ML Crankset, )

// Modified: (951, HL Crankset, 8) => (951, HL Crankset, )

}

Enable and disable tracking

DbContext’s default behavior is to track all changes automatically. This can be turned off if not needed. To disable tracking for specific entities queried from repository, call the EntityFrameworkQueryableExtensions.AsNoTracking extension method for IQueryable<T> query:

internal static void AsNoTracking(AdventureWorks adventureWorks)

Product untracked = adventureWorks.Products.AsNoTracking().First();

adventureWorks.ChangeTracker.Entries().Count().WriteLine(); // 0

}

Tracking can also be enabled or disabled at the DbContext scope, by setting the ChangeTracker.AutoDetectChangesEnabled property to true or false. The default value of ChangeTracker.AutoDetectChangesEnabled is true, so usually it is not needed to manually detect changes by calling ChangeTracker.DetectChanges method. The changes are automatically detected when DbContext.SubmitChanges is called. The changes are also automatically detected when tracking information is calculated, for example, when calling ChangeTracker.Entries, DbContext.Entry, etc.

If needed, changes and be manually tracked by calling ChangeTracker.DetectChanges method:

internal static void DetectChanges(AdventureWorks adventureWorks)

adventureWorks.ChangeTracker.AutoDetectChangesEnabled = false;

Product product = adventureWorks.Products.First();

product.ListPrice += 100;

adventureWorks.ChangeTracker.HasChanges().WriteLine(); // False

adventureWorks.ChangeTracker.DetectChanges();

adventureWorks.ChangeTracker.HasChanges().WriteLine(); // True

}

Change data

To change the data in the database, just create a DbContext instance, change the data in its repositories, and call DbContext.SaveChanges method to submit the tracked changes to the remote database as a unit of work.

Create

To create new entities into the repository, call DbSet<T>.Add or DbSet<T>.AddRange. The following example creates a new category, and a new related subcategory, and add to repositories:

internal static ProductCategory Create()

using (AdventureWorks adventureWorks = new AdventureWorks())

ProductCategory category = new ProductCategory() { Name = "Create" };

ProductSubcategory subcategory = new ProductSubcategory() { Name = "Create" };

category.ProductSubcategories = new HashSet<ProductSubcategory>() { subcategory };

// Equivalent to: subcategory.ProductCategory = category;

category.ProductCategoryID.WriteLine(); // 0

subcategory.ProductCategoryID.WriteLine(); // 0

subcategory.ProductSubcategoryID.WriteLine(); // 0

adventureWorks.ProductCategories.Add(category); // Track creation.

// Equivalent to: adventureWorks.ProductSubcategories.Add(subcategory);

adventureWorks.ChangeTracker.Entries()

.Count(tracking => tracking.State == EntityState.Added).WriteLine(); // 2

object.ReferenceEquals(category.ProductSubcategories.Single(), subcategory).WriteLine(); // True

adventureWorks.SaveChanges().WriteLine(); // 2

// BEGIN TRANSACTION

// exec sp_executesql N'SET NOCOUNT ON;

// INSERT INTO [Production].[ProductCategory] ([Name])

// VALUES (@p0);

// SELECT [ProductCategoryID]

// FROM [Production].[ProductCategory]

// WHERE @@ROWCOUNT = 1 AND [ProductCategoryID] = scope_identity();

// ',N'@p0 nvarchar(50)',@p0=N'Create'

//

// exec sp_executesql N'SET NOCOUNT ON;

// INSERT INTO [Production].[ProductCategory] ([Name])

// VALUES (@p0);

// SELECT [ProductCategoryID]

// FROM [Production].[ProductCategory]

// WHERE @@ROWCOUNT = 1 AND [ProductCategoryID] = scope_identity();

// ',N'@p0 nvarchar(50)',@p0=N'Create'

// COMMIT TRANSACTION

adventureWorks.ChangeTracker.Entries()

.Count(tracking => tracking.State != EntityState.Unchanged).WriteLine(); // 0

category.ProductCategoryID.WriteLine(); // 5

subcategory.ProductCategoryID.WriteLine(); // 5

subcategory.ProductSubcategoryID.WriteLine(); // 38

return category;

} // Unit of work.

}

Here DbSet<T>.Add is called only once with 1 subcategory entity. Internally, Add triggers change detection, and tracks this subcategory as Added state. Since this subcategory is related with another category entity with navigation property, the related category is also tracked, as the Added state too. So in total there are 2 entity changes tracked. When DbContext.SaveChanges is called, EF Core translates these 2 changes to 2 SQL INSERT statements:

The category’s key is identity key, with value generated by database, so is subcategory. So in the translated INSERT statements, the new category’s ProductCategoryID and the new subcategory’s ProductSubcategory are ignored. After the each new row is created, a SELECT statement calls SCOPE_IDENTITY metadata function to read the last generated identity value, which is the primary key of the inserted row. As a result, since there are 2 row changes in total, SaveChanges returns 2, And the 2 changes are submitted in a transaction, so that all changes can succeed or fail as a unit.

DbSet<T>.AddRange can be called with multiple entities. AddRange only triggers change detection once for all the entities, so it can have better performance than multiple Add calls,

Update

To update entities in the repositories, just change their properties, including navigation properties. The following example updates a subcategory entity’s name, and related category entity, which is translated to UPDATE statement:

internal static void Update(int categoryId, int subcategoryId)

using (AdventureWorks adventureWorks = new AdventureWorks())

ProductCategory category = adventureWorks.ProductCategories.Find(categoryId);

ProductSubcategory subcategory = adventureWorks.ProductSubcategories.Find(subcategoryId);

$"({subcategory.ProductSubcategoryID}, {subcategory.Name}, {subcategory.ProductCategoryID})"

.WriteLine(); // (48, Create, 25)

subcategory.Name = "Update"; // Entity property update.

subcategory.ProductCategory = category; // Relashionship (foreign key) update.

adventureWorks.ChangeTracker.Entries().Count(tracking => tracking.State != EntityState.Unchanged)

.WriteLine(); // 1

$"({subcategory.ProductSubcategoryID}, {subcategory.Name}, {subcategory.ProductCategoryID})"

.WriteLine(); // (48, Update, 1)

adventureWorks.SaveChanges().WriteLine(); // 1

// BEGIN TRANSACTION

// exec sp_executesql N'SET NOCOUNT ON;

// UPDATE [Production].[ProductSubcategory] SET [Name] = @p0, [ProductCategoryID] = @p1

// WHERE [ProductSubcategoryID] = @p2;

// SELECT @@ROWCOUNT;

// ',N'@p2 int,@p0 nvarchar(50),@p1 int',@p2=25,@p0=N'Update',@p1=25

// COMMIT TRANSACTION

} // Unit of work.

}

The above example first call Find to read the entities with a SELECT query, then execute the UPDATE statement. Here the row to update is located by primary key, so, if the primary key is known, then it can be used directly:

internal static void UpdateWithoutRead(int categoryId)

using (AdventureWorks adventureWorks = new AdventureWorks())

ProductCategory category = new ProductCategory()

ProductCategoryID = categoryId,

Name = Guid.NewGuid().ToString() // To be updated.

};

adventureWorks.ProductCategories.Attach(category); // Track entity.

EntityEntry tracking = adventureWorks.ChangeTracker.Entries<ProductCategory>().Single();

tracking.State.WriteLine(); // Unchanged

tracking.State = EntityState.Modified;

adventureWorks.SaveChanges().WriteLine(); // 1

// BEGIN TRANSACTION

// exec sp_executesql N'SET NOCOUNT ON;

// UPDATE [Production].[ProductCategory] SET [Name] = @p0

// WHERE [ProductCategoryID] = @p1;

// SELECT @@ROWCOUNT;

// ',N'@p1 int,@p0 nvarchar(50)',@p1=25,@p0=N'513ce396-4a5e-4a86-9d82-46f284aa4f94'

// COMMIT TRANSACTION

} // Unit of work.

}

Here a category entity is constructed on the fly, with specified primary key and updated Name. To track and save the changes, ii is attached to the repository. As fore mentioned, the attached entity is tracked as Unchanged state, so just manually set its state to Modified. This time, only one UPDATE statement is translated and executed, without SELECT.

When there is no change to save, SaveChanges does not translate or execute any SQL and returns 0:

internal static void SaveNoChanges(int categoryId)

using (AdventureWorks adventureWorks = new AdventureWorks())

ProductCategory category = adventureWorks.ProductCategories.Find(categoryId);

string originalName = category.Name;

category.Name = Guid.NewGuid().ToString(); // Entity property update.

category.Name = originalName; // Entity property update.

EntityEntry tracking = adventureWorks.ChangeTracker.Entries().Single();

tracking.State.WriteLine(); // Unchanged

adventureWorks.ChangeTracker.HasChanges().WriteLine(); // False

adventureWorks.SaveChanges().WriteLine(); // 0

} // Unit of work.

}

Delete

To delete entities from the repositories, call DbSet<T>.Remove or DbSet<T>.RemoveRange. The following example read an entity then delete it:

internal static void Delete(int subcategoryId)

using (AdventureWorks adventureWorks = new AdventureWorks())

ProductSubcategory subcategory = adventureWorks.ProductSubcategories.Find(subcategoryId);

adventureWorks.ChangeTracker.Entries().Count().WriteLine(); // 1

adventureWorks.ChangeTracker.Entries<ProductSubcategory>().Single().State.WriteLine(); // Unchanged

adventureWorks.ProductSubcategories.Remove(subcategory); // Track deletion.

adventureWorks.ChangeTracker.Entries<ProductSubcategory>().Single().State.WriteLine(); // Deleted

adventureWorks.SaveChanges().WriteLine(); // 1

// BEGIN TRANSACTION

// exec sp_executesql N'SET NOCOUNT ON;

// DELETE FROM [Production].[ProductSubcategory]

// WHERE [ProductSubcategoryID] = @p0;

// SELECT @@ROWCOUNT;

// ',N'@p0 int',@p0=48

// COMMIT TRANSACTION

} // Unit of work.

}

Here, the row to delete is also located with primary key. So again, when primary key is known, reading entity can be skipped:

internal static void DeleteWithoutRead(int categoryId)

using (AdventureWorks adventureWorks = new AdventureWorks())

ProductCategory category = new ProductCategory() { ProductCategoryID = categoryId };

adventureWorks.ProductCategories.Attach(category);

adventureWorks.ChangeTracker.Entries().Count().WriteLine(); // 1

adventureWorks.ChangeTracker.Entries<ProductCategory>().Single().State.WriteLine(); // Unchanged

adventureWorks.ProductCategories.Remove(category); // Track deletion.

adventureWorks.ChangeTracker.Entries<ProductCategory>().Single().State.WriteLine(); // Deleted

adventureWorks.SaveChanges().WriteLine(); // 1

// BEGIN TRANSACTION

// exec sp_executesql N'SET NOCOUNT ON;

// DELETE FROM [Production].[ProductCategory]

// WHERE [ProductCategoryID] = @p0;

// SELECT @@ROWCOUNT;

// ',N'@p0 int',@p0=25

// COMMIT TRANSACTION

} // Unit of work.

}

If a principal entity is loaded with its dependent entities, deleting the principal entity becomes cascade deletion:

internal static void DeleteCascade(int categoryId)

using (AdventureWorks adventureWorks = new AdventureWorks())

ProductCategory category = adventureWorks.ProductCategories

.Include(entity => entity.ProductSubcategories)

.Single(entity => entity.ProductCategoryID == categoryId);

ProductSubcategory subcategory = category.ProductSubcategories.Single();

adventureWorks.ChangeTracker.Entries().Count().WriteLine(); // 2

adventureWorks.ProductCategories.Remove(category); // Track deletion.

// Optional: adventureWorks.ProductSubcategories.Remove(subcategory);

adventureWorks.ChangeTracker.Entries().Count(tracking => tracking.State == EntityState.Deleted)

.WriteLine(); // 2

adventureWorks.SaveChanges().WriteLine(); // 2

// BEGIN TRANSACTION

// exec sp_executesql N'SET NOCOUNT ON;

// DELETE FROM [Production].[ProductSubcategory]

// WHERE [ProductSubcategoryID] = @p0;

// SELECT @@ROWCOUNT;

// ',N'@p0 int',@p0=49

// exec sp_executesql N'SET NOCOUNT ON;

// DELETE FROM [Production].[ProductCategory]

// WHERE [ProductCategoryID] = @p1;

// SELECT @@ROWCOUNT;

// ',N'@p1 int',@p1=26

// COMMIT TRANSACTION

} // Unit of work.

}

Here the cascade deletion are translated and executed in the right order. The subcategory is deleted first, then category is deleted.

Transaction

As discussed above, by default DbContext.SaveChanges execute all data creation, update and deletion in a transaction, so that all the work can succeed or fail as a unit. If the unit of work succeeds, the transaction is committed, if any operation fails, the transaction is rolled back. EF Core also supports custom transactions.

Transaction with connection resiliency and execution strategy

If the retry strategy is enabled for connection resiliency for DbContext by default, then this default retry strategy does not work custom transaction. Custom transaction works within a single retry operation, but not cross multiple retries. In EF Core, database façade’s CreateExecutionStrategy method can be called to explicitly specify a single retry operation:

internal static void ExecutionStrategy(AdventureWorks adventureWorks)

adventureWorks.Database.CreateExecutionStrategy().Execute(() =>

// Single retry operation, which can have custom transactions.

});

}

EF Core transaction

EF Core provides Microsoft.EntityFrameworkCore.Storage.IDbContextTransaction to represent a transaction. It can be created by DbContext.Database.BeginTransaction, where the transaction’s isolation level can be optionally specified. The following example executes a entity change and custom SQL with one EF Core transaction:

internal static void DbContextTransaction(AdventureWorks adventureWorks)

adventureWorks.Database.CreateExecutionStrategy().Execute(() =>

using (IDbContextTransaction transaction = adventureWorks.Database

.BeginTransaction(IsolationLevel.ReadUncommitted))

try

ProductCategory category = new ProductCategory() { Name = nameof(ProductCategory) };

adventureWorks.ProductCategories.Add(category);

adventureWorks.SaveChanges().WriteLine(); // 1

adventureWorks.Database

.ExecuteSqlCommand($@"DELETE FROM [Production].[ProductCategory] WHERE [Name] = {nameof(ProductCategory)}")

.WriteLine(); // 1

adventureWorks.CurrentIsolationLevel().WriteLine(); // ReadUncommitted transaction.Commit();

catch

transaction.Rollback();

throw;

});

}

EF Core transaction wraps ADO.NET transaction. When the EF Core transaction begins, The specified isolation level is written to a packet (represented by System.Data.SqlClient.SNIPacket type), and sent to SQL database via TDS protocol. There is no SQL statement like SET TRANSACTION ISOLATION LEVEL executed, so the actual isolation level cannot be logged by EF Core, or traced by SQL Profiler. In above example, CurrentIsolationLevel is called to verify the current transaction’s isolation level. It is an extension method of DbContext. It queries the dynamic management view sys.dm_exec_sessions with current session id, which can be retrieved with @@SPID function:

internal static IsolationLevel CurrentIsolationLevel(this DbConnection connection,

DbTransaction transaction = null)

using (DbCommand command = connection.CreateCommand())

command.CommandText =

@"SELECT transaction_isolation_level FROM sys.dm_exec_sessions WHERE session_id = @@SPID";

command.Transaction = transaction;

switch ((short)command.ExecuteScalar())

case 0: return IsolationLevel.Unspecified;

case 1: return IsolationLevel.ReadUncommitted;

case 2: return IsolationLevel.ReadCommitted;

case 3: return IsolationLevel.RepeatableRead;

case 4: return IsolationLevel.Serializable;

case 5: return IsolationLevel.Snapshot;

default: throw new InvalidOperationException();

internal static IsolationLevel CurrentIsolationLevel(this DbContext dbContext) =>

dbContext.Database.GetDbConnection().CurrentIsolationLevel(

dbContext.Database.CurrentTransaction?.GetDbTransaction());

When DbContext.SaveChanges is called to create entity. it detects a transaction is explicitly created with the current DbContext, so it uses that transaction and does not automatically begins a new transaction like all the previous examples. Then DbContext.Database.ExecuteSqlCommnd is called to delete entity. It also detects and uses transaction of the current DbContext. Eventually, to commit the transaction, call IDbContextTransaction.Commit, to rollback the transaction, call IDbContextTransaction.Rollback.

ADO.NET transaction

EF Core can also use the ADO.NET transaction, represented by System.Data.Common.DbTransaction. The following example execute the same entity change and custom SQL command with one ADO.NET transaction. To use an existing ADO.NET transaction, call DbContext.Database.UseTransaction:

internal static void DbTransaction()

using (DbConnection connection = new SqlConnection(ConnectionStrings.AdventureWorks))

connection.Open();

using (DbTransaction transaction = connection.BeginTransaction(IsolationLevel.RepeatableRead))

try

using (AdventureWorks adventureWorks = new AdventureWorks(connection))

adventureWorks.Database.CreateExecutionStrategy().Execute(() =>

adventureWorks.Database.UseTransaction(transaction);

adventureWorks.CurrentIsolationLevel().WriteLine(); // RepeatableRead

ProductCategory category = new ProductCategory() { Name = nameof(ProductCategory) };

adventureWorks.ProductCategories.Add(category);

adventureWorks.SaveChanges().WriteLine(); // 1.

});

using (DbCommand command = connection.CreateCommand())

command.CommandText = "DELETE FROM [Production].[ProductCategory] WHERE [Name] = @Name";

DbParameter parameter = command.CreateParameter();

parameter.ParameterName = "@Name";

parameter.Value = nameof(ProductCategory);

command.Parameters.Add(parameter);

command.Transaction = transaction;

command.ExecuteNonQuery().WriteLine(); // 1

connection.CurrentIsolationLevel(transaction).WriteLine(); // RepeatableRead

transaction.Commit();

catch

transaction.Rollback();

throw;

}

Transaction scope

As fore mentioned, EF Core transaction only works with its source DbContext, and the ADO.NET transaction only work with its source DbConnection. EF Core can also use System.Transactions.TransactionScope to have a transaction that work across the lifecycle of multiple DbContext or DbConnection instances:

internal static void TransactionScope(AdventureWorks adventureWorks)

adventureWorks.Database.CreateExecutionStrategy().Execute(() =>

using (TransactionScope scope = new TransactionScope(

TransactionScopeOption.Required,

new TransactionOptions() { IsolationLevel = IsolationLevel.Serializable }))

using (DbConnection connection = new SqlConnection(ConnectionStrings.AdventureWorks))

using (DbCommand command = connection.CreateCommand())

command.CommandText = "INSERT INTO [Production].[ProductCategory] ([Name]) VALUES(@Name); ";

DbParameter parameter = command.CreateParameter();

parameter.ParameterName = "@Name";

parameter.Value = nameof(ProductCategory);

command.Parameters.Add(parameter);

connection.Open();

command.ExecuteNonQuery().WriteLine(); // 1

connection.CurrentIsolationLevel().WriteLine(); // Serializable

using (AdventureWorks adventureWorks1 = new AdventureWorks())

ProductCategory category = adventureWorks1.ProductCategories

.Single(entity => entity.Name == nameof(ProductCategory));

adventureWorks1.ProductCategories.Remove(category);

adventureWorks1.SaveChanges().WriteLine(); // 1

adventureWorks1.CurrentIsolationLevel().WriteLine(); // Serializable

scope.Complete();

});

}

Resolving optimistic concurrency

Conflicts can occur if the same data is read and changed concurrently. Generally, there are 2 concurrency control approaches:

· Pessimistic concurrency: one database client can lock the data being accessed, in order to prevent other database clients to change that same data concurrently.

· Optimistic concurrency: Data is not locked in the database for client to CRUD. Any database client is allowed to read and change any data concurrently. As a result, concurrency conflicts can happen. This is how EF Core work with database.

To demonstrate the behavior of EF Core for concurrency, the following DbReaderWriter type is defined as database CRUD client:

internal partial class DbReaderWriter : IDisposable

private readonly DbContext context;

internal DbReaderWriter(DbContext context) => this.context = context;

internal TEntity Read<TEntity>(params object[] keys) where TEntity : class =>

this.context.Set<TEntity>().Find(keys);

internal int Write(Action change)

change();

return this.context.SaveChanges();

internal DbSet<TEntity> Set<TEntity>() where TEntity : class => this.context.Set<TEntity>();

public void Dispose() => this.context.Dispose();

}

Multiple DbReaderWriter instances can be be used to read and write data concurrently. For example:

internal static void NoCheck(

DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3)

int id = 1;

ProductCategory categoryCopy1 = readerWriter1.Read<ProductCategory>(id);

ProductCategory categoryCopy2 = readerWriter2.Read<ProductCategory>(id);

readerWriter1.Write(() => categoryCopy1.Name = nameof(readerWriter1));

// exec sp_executesql N'SET NOCOUNT ON;

// UPDATE [Production].[ProductCategory] SET [Name] = @p0

// WHERE [ProductCategoryID] = @p1;

// SELECT @@ROWCOUNT;

// ',N'@p1 int,@p0 nvarchar(50)',@p1=1,@p0=N'readerWriter1'

readerWriter2.Write(() => categoryCopy2.Name = nameof(readerWriter2)); // Last client wins.

// exec sp_executesql N'SET NOCOUNT ON;

// UPDATE [Production].[ProductCategory] SET [Name] = @p0

// WHERE [ProductCategoryID] = @p1;

// SELECT @@ROWCOUNT;

// ',N'@p1 int,@p0 nvarchar(50)',@p1=1,@p0=N'readerWriter2'

ProductCategory category3 = readerWriter3.Read<ProductCategory>(id);

category3.Name.WriteLine(); // readerWriter2

}

In this example, multiple DbReaderWriter instances read and write data concurrently:

readerWriter1 reads category “Bikes”
readerWriter2 reads category “Bikes”. These 2 entities are independent because they are are from different DbContext instances.
readerWriter1 updates category’s name from “Bikes” to “readerWriter1”. As previously discussed, by default EF Core locate the category with its primary key.
In database, this category’s name is no longer “Bikes”
readerWriter2 updates category’s name from “Bikes” to “readerWriter2”. It locates the category with its primary key as well. The primary key is unchanged, so the same category can be located and the name can be changed.
So later when readerWriter3 reads the entity with the same primary key, the category entity’s Name is “readerWriter2”.

Detect Concurrency conflicts

Concurrency conflicts can be detected by checking entities’ property values besides primary keys. To required EF Core to check a certain property, just add a System.ComponentModel.DataAnnotations.ConcurrencyCheckAttribute to it. Remember when defining ProductPhoto entity, its ModifiedDate has a [ConcurrencyCheck] attribute:

public partial class ProductPhoto

[ConcurrencyCheck]

public DateTime ModifiedDate { get; set; }

}

This property is also called the concurrency token. When EF Core translate changes of a photo, ModifiedDate property is checked along with the primary key to locate the photo:

internal static void ConcurrencyCheck(DbReaderWriter readerWriter1, DbReaderWriter readerWriter2)

int id = 1;

ProductPhoto photoCopy1 = readerWriter1.Read<ProductPhoto>(id);

ProductPhoto photoCopy2 = readerWriter2.Read<ProductPhoto>(id);

readerWriter1.Write(() =>

photoCopy1.LargePhotoFileName = nameof(readerWriter1);

photoCopy1.ModifiedDate = DateTime.Now;

});

// exec sp_executesql N'SET NOCOUNT ON;

// UPDATE [Production].[ProductPhoto] SET [LargePhotoFileName] = @p0, [ModifiedDate] = @p1

// WHERE [ProductPhotoID] = @p2 AND [ModifiedDate] = @p3;

// SELECT @@ROWCOUNT;

// ',N'@p2 int,@p0 nvarchar(50),@p1 datetime2(7),@p3 datetime2(7)',@p2=1,@p0=N'readerWriter1',@p1='2017-01-25 22:04:25.9292433',@p3='2008-04-30 00:00:00'

readerWriter2.Write(() =>

photoCopy2.LargePhotoFileName = nameof(readerWriter2);

photoCopy2.ModifiedDate = DateTime.Now;

});

// exec sp_executesql N'SET NOCOUNT ON;

// UPDATE [Production].[ProductPhoto] SET [LargePhotoFileName] = @p0, [ModifiedDate] = @p1

// WHERE [ProductPhotoID] = @p2 AND [ModifiedDate] = @p3;

// SELECT @@ROWCOUNT;

// ',N'@p2 int,@p0 nvarchar(50),@p1 datetime2(7),@p3 datetime2(7)',@p2=1,@p0=N'readerWriter2',@p1='2017-01-25 22:04:59.1792263',@p3='2008-04-30 00:00:00'

// DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded.

}

In the translated SQL statement, the WHERE clause contains primary key and the original concurrency token. The following is how EF Core check the concurrency conflicts:

readerWriter1 reads photo with primary key 1, and modified date “2008-04-30 00:00:00”
readerWriter2 reads the same photo with primary key 1, and modified date “2008-04-30 00:00:00”
readerWriter1 locates the photo with primary key and original modified date, and update its large photo file name and modified date.
In database the photo’s modified date is no longer the original value “2008-04-30 00:00:00”
readerWriter2 tries to locate the photo with primary key and original modified date. However the provided modified date is outdated. EF Core detect that 0 row is updated by the translated SQL, and throws DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded.

Microsoft’s AdventureWorks sample database does not have such a rowversion column, so create one for the Production.Product table:

ALTER TABLE [Production].[Product] ADD [RowVersion] rowversion NOT NULL

Then define the mapping property for Product entity:

public partial class Product

[DatabaseGenerated(DatabaseGeneratedOption.Computed)]

[Timestamp]

public byte[] RowVersion { get; set; }

[NotMapped]

public string RowVersionString =>

$"0x{BitConverter.ToUInt64(this.RowVersion.Reverse().ToArray(), 0).ToString("X16")}";

}

internal static void RowVersion(DbReaderWriter readerWriter1, DbReaderWriter readerWriter2)

int id = 995;

Product productCopy1 = readerWriter1.Read<Product>(id);

productCopy1.RowVersionString.WriteLine(); // 0x0000000000000803

Product productCopy2 = readerWriter2.Read<Product>(id);

productCopy2.RowVersionString.WriteLine(); // 0x0000000000000803

readerWriter1.Write(() => productCopy1.Name = nameof(readerWriter1));

// exec sp_executesql N'SET NOCOUNT ON;

// UPDATE [Production].[Product] SET [Name] = @p0

// WHERE [ProductID] = @p1 AND [RowVersion] = @p2;

// SELECT [RowVersion]

// FROM [Production].[Product]

// WHERE @@ROWCOUNT = 1 AND [ProductID] = @p1;

// ',N'@p1 int,@p0 nvarchar(50),@p2 varbinary(8)',@p1=995,@p0=N'readerWriter1',@p2=0x0000000000000803

productCopy1.RowVersionString.WriteLine(); // 0x00000000000324B1

readerWriter2.Write(() => readerWriter2.Set<Product>().Remove(productCopy2));

// exec sp_executesql N'SET NOCOUNT ON;

// DELETE FROM [Production].[Product]

// WHERE [ProductID] = @p0 AND [RowVersion] = @p1;

// SELECT @@ROWCOUNT;

// ',N'@p0 int,@p1 varbinary(8)',@p0=995,@p1=0x0000000000000803

// DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded.

}

When updating and deleting photo entities, its auto generated RowVersion property value is checked too. So this is how it works:

readerWriter1 reads product with primary key 995 and row version 0x0000000000000803
readerWriter2 reads product with the same primary key 995 and row version 0x0000000000000803
readerWriter1 locates the photo with primary key and original row version, and update its name. Database automatically increases the photo’s row version. Since the row version is specified as [DatabaseGenerated(DatabaseGeneratedOption.Computed)], EF Core also locate the photo with the primary key to query the increased row version, and update the entity at client side.
In database the product’s row version is no longer 0x0000000000000803.
Then readerWriter2 tries to locate the product with primary key and original row version, and delete it. No product can be found with outdated row version, EF Core detect that 0 row is deleted, and throws DbUpdateConcurrencyException.

Resolve concurrency conflicts

DbUpdateConcurrencyException is thrown when SaveChanges detects concurrency conflict:

namespace Microsoft.EntityFrameworkCore

public class DbUpdateException : Exception

public virtual IReadOnlyList<EntityEntry> Entries { get; }

// Other members.

public class DbUpdateConcurrencyException : DbUpdateException

// Members.

}

internal partial class DbReaderWriter

internal int Write(Action change, Action<DbUpdateConcurrencyException> handleException, int retryCount = 3)

change();

for (int retry = 1; retry < retryCount; retry++)

try

return this.context.SaveChanges();

catch (DbUpdateConcurrencyException exception)

handleException(exception);

return this.context.SaveChanges();

}

Retain database values (database wins)

Similar to previous examples, the following example has multiple DbReaderWriter instances to update a product concurrently:

internal static void UpdateProduct(

DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3,

Action<EntityEntry>resolveConflicts)

int id = 950;

Product productCopy1 = readerWriter1.Read<Product>(id);

Product productCopy2 = readerWriter2.Read<Product>(id);

readerWriter1.Write(() =>

productCopy1.Name = nameof(readerWriter1);

productCopy1.ListPrice = 100.0000M;

});

readerWriter2.Write(

change: () =>

productCopy2.Name = nameof(readerWriter2);

productCopy2.ProductSubcategoryID = 1;

},

handleException: exception =>

EntityEntry tracking = exception.Entries.Single();

Product original = (Product)tracking.OriginalValues.ToObject();

Product current = (Product)tracking.CurrentValues.ToObject();

Product database = productCopy1; // Values saved in database.

$"Original: ({original.Name}, {original.ListPrice}, {original.ProductSubcategoryID}, {original.RowVersionString})"

.WriteLine();

$"Database: ({database.Name}, {database.ListPrice}, {database.ProductSubcategoryID}, {database.RowVersionString})"

.WriteLine();

$"Update to: ({current.Name}, {current.ListPrice}, {current.ProductSubcategoryID})"

.WriteLine();

resolveConflicts(tracking);

});

Product resolved = readerWriter3.Read<Product>(id);

$"Resolved: ({resolved.Name}, {resolved.ListPrice}, {resolved.ProductSubcategoryID}, {resolved.RowVersionString})"

.WriteLine();

}

This is how it works with concurrency conflicts:

readerWriter1 reads product with primary key 950, and RowVersion 0x00000000000007D1
readerWriter2 reads product with the same primary key 950, and RowVersion 0x00000000000007D1
readerWriter1 locates product with primary key and original RowVersion 0x00000000000007D1, and updates product’s name and list price. Database automatically increases the product’s row version
In database the product’s row version is no longer 0x00000000000007D1.
readerWriter2 tries to locate product with primary key and original RowVersion, and update product’s name and subcategory.
readerWriter2 fails to update product, because it cannot locate the product with original RowVersion 0x00000000000007D1. Again, no product can be found with outdated row version, DbUpdateConcurrencyException is thrown.

· product’s original property values read by readerWriter2 before the changes

· product’s property values in database at this moment, which are already updated readerWriter1

· product’s current property values after changes, which readerWriter2 fails to save to database.

internal partial class DbReaderWriter

internal int WriteDatabaseWins(Action change)

change();

try

return this.context.SaveChanges();

catch (DbUpdateConcurrencyException)

return 0; // this.context is in a corrupted state.

}

internal static void DatabaseWins(

DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3)

UpdateProduct(readerWriter1, readerWriter2, readerWriter3, resolveConflicts: tracking =>

tracking.State.WriteLine(); // Modified

tracking.Property(nameof(Product.Name)).IsModified.WriteLine(); // True

tracking.Property(nameof(Product.ListPrice)).IsModified.WriteLine(); // False

tracking.Property(nameof(Product.ProductSubcategoryID)).IsModified.WriteLine(); // True

tracking.Reload(); // Execute query.

tracking.State.WriteLine(); // Unchanged

tracking.Property(nameof(Product.Name)).IsModified.WriteLine(); // False

tracking.Property(nameof(Product.ListPrice)).IsModified.WriteLine(); // False

tracking.Property(nameof(Product.ProductSubcategoryID)).IsModified.WriteLine(); // False

});

// Original: (ML Crankset, 256.4900, 8, 0x00000000000007D1)

// Database: (readerWriter1, 100.0000, 8, 0x0000000000036335)

// Update to: (readerWriter2, 256.4900, 1)

// Resolved: (readerWriter1, 100.0000, 8, 0x0000000000036335)

}

EntityEntry.Reload executes a SELECT statement to read the product’s property values from database, then refresh the product entity and all tracking information. The product’s property values, the tracked original property values before changes, the tracked current property values after changes, are all refreshed to the queried database values. The entity tracking state is also refreshed to Unchanged.
At this moment, product has the same tracked original values and current values, as if it is just initially read from database, without changes.
When DbReaderWriter.Write’s retry logic calls SaveChanges again, no changed entity is detected. SaveChanges succeeds without executing any SQL, and returns 0. As expected, readerWriter2 does not update any value to database, and all values in database are retained.

Later, when readerWriter3 reads the product again, product has all values updated by readerWrtier1.

Overwrite database values (client wins)

Another simple option, called “client wins”, is to disregard values in database, and overwrite them with whatever data submitted from client.

internal static void ClientWins(

DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3)

UpdateProduct(readerWriter1, readerWriter2, readerWriter3, resolveConflicts: tracking =>

PropertyValues databaseValues = tracking.GetDatabaseValues();

// Refresh original values, which go to WHERE clause of UPDATE statement.

tracking.OriginalValues.SetValues(databaseValues);

tracking.State.WriteLine(); // Modified

tracking.Property(nameof(Product.Name)).IsModified.WriteLine(); // True

tracking.Property(nameof(Product.ListPrice)).IsModified.WriteLine(); // True

tracking.Property(nameof(Product.ProductSubcategoryID)).IsModified.WriteLine(); // True

});

// Original: (ML Crankset, 256.4900, 8, 0x00000000000007D1)

// Database: (readerWriter1, 100.0000, 8, 0x0000000000036336)

// Update to: (readerWriter2, 256.4900, 1)

// Resolved: (readerWriter2, 256.4900, 1, 0x0000000000036337)

}

The same conflict is resolved differently:

EntityEntry.GetDatabaseValues executes a SELECT statement to read the product’s property values from database, including the updated row version. This call does not impact the product values or tracking information.
Manually set the tracked original property values to the queried database values. The entity tracking state is still Changed. The original property values become all different from tracked current property values. So all product properties are tracked as modified.
At this moment, the product has tracked original values updated, and keeps all tracked current values, as if it is read from database after readerWriter1 updates the name and list price, and then have all properties values changed.
When DbReaderWriter.Write’s retry logic calls SaveChanges again, product changes are detected to submit. So EF Core translate the product change to a UPDATE statement. In the SET clause, since there are 3 properties tracked as modified, 3 columns are set. In the WHERE clause, to locate the product, the tracked original row version has been set to the updated value from database. This time product can be located, and all 3 properties are updated. SaveChanges succeeds and returns 1. As expected, readerWriter2 updates all value to database.

Later, when readerWriter3 reads the product again, product has all values updated by readerWrter2.

Merge with database values

A more complex but useful option, is to merge the client values and database values. For each property:

· If original value is different from database value, which means database value is already updated by other concurrent client, then give up updating this property, and retain the database value

· If original value is the same as database value, which means no concurrency conflict for this property, then process normally to submit the change

internal static void MergeClientAndDatabase(

DbReaderWriter readerWriter1, DbReaderWriter readerWriter2, DbReaderWriter readerWriter3)

UpdateProduct(readerWriter1, readerWriter2, readerWriter3, resolveConflicts: tracking =>

PropertyValues databaseValues = tracking.GetDatabaseValues(); // Execute query.

PropertyValues originalValues = tracking.OriginalValues.Clone();

// Refresh original values, which go to WHERE clause.

tracking.OriginalValues.SetValues(databaseValues);

// If database has an different value for a property, then retain the database value.

databaseValues.Properties // Navigation properties are not included.

.Where(property => !object.Equals(originalValues[property.Name], databaseValues[property.Name]))

.ForEach(property => tracking.Property(property.Name).IsModified = false);

tracking.State.WriteLine(); // Modified

tracking.Property(nameof(Product.Name)).IsModified.WriteLine(); // False

tracking.Property(nameof(Product.ListPrice)).IsModified.WriteLine(); // False

tracking.Property(nameof(Product.ProductSubcategoryID)).IsModified.WriteLine(); // True

});

// Original: (ML Crankset, 256.4900, 8, 0x00000000000007D1)

// Database: (readerWriter1, 100.0000, 8, 0x0000000000036338)

// Update to: (readerWriter2, 256.4900, 1)

// Resolved: (readerWriter1, 100.0000, 1, 0x0000000000036339)

}

With this approach:

Again, EntityEntry.GetDatabaseValues executes a SELECT statement to read the product’s property values from database, including the updated row version.
Backup tracked original values, then refresh conflict.OriginalValues to the database values, so that these values can go to the translated WHERE clause. Again, the entity tracking state is still Changed. The original property values become all different from tracked current property values. So all product values are tracked as modified and should go to SET clause.
For each property, if the backed original value is different from the database value, it means this property is changed by other client and there is concurrency conflict. In this case, revert this property’s tracking status to unmodified. The name and list price are reverted.
At this moment, the product has tracked original values updated, and only keeps tracked current value of subcategory, as if it is read from database after readerWriter1 updates the name and list price, and then only have subcategory changed, which has no conflict.
When DbReaderWriter.Write’s retry logic calls SaveChanges again, product changes are detected to submit. Here only subcategory is updated to database. SaveChanges succeeds and returns 1. As expected, readerWriter2 only updates value without conflict, the other conflicted values are retained.

Later, when readerWriter3 reads the product, product has name and list price values updated by readerWrtier1, and has subcategory updated by readerWriter2.

Save changes with concurrency conflict handling

Similar to above DbReaderWriter.Write method, a general SaveChanges extension method for DbContext can be defined to handle concurrency conflicts and apply simple retry logic:

public static int SaveChanges(

this DbContext context, Action<IEnumerable<EntityEntry>> resolveConflicts, int retryCount = 3)

if (retryCount <= 0)

throw new ArgumentOutOfRangeException(nameof(retryCount));

for (int retry = 1; retry < retryCount; retry++)

try

return context.SaveChanges();

catch (DbUpdateConcurrencyException exception) when (retry < retryCount)

resolveConflicts(exception.Entries);

return context.SaveChanges();

}

public class TransientDetection<TException> : ITransientErrorDetectionStrategy

where TException : Exception

public bool IsTransient(Exception ex) => ex is TException;

public static int SaveChanges(

this DbContext context, Action<IEnumerable<EntityEntry>> resolveConflicts, RetryStrategy retryStrategy)

RetryPolicy retryPolicy = new RetryPolicy(

errorDetectionStrategy: new TransientDetection<DbUpdateConcurrencyException>(),

retryStrategy: retryStrategy);

retryPolicy.Retrying += (sender, e) =>

resolveConflicts(((DbUpdateConcurrencyException)e.LastException).Entries);

return retryPolicy.ExecuteAction(context.SaveChanges);

}

public enum RefreshConflict

StoreWins,

ClientWins,

MergeClientAndStore

public static int SaveChanges(this DbContext context, RefreshConflict refreshMode, int retryCount = 3)

if (retryCount< = 0)

throw new ArgumentOutOfRangeException(nameof(retryCount));

return context.SaveChanges(

conflicts => conflicts.ForEach(tracking => tracking.Refresh(refreshMode)), retryCount);

public static int SaveChanges(

this DbContext context, RefreshConflict refreshMode, RetryStrategy retryStrategy) =>

context.SaveChanges(

conflicts => conflicts.ForEach(tracking => tracking.Refresh(refreshMode)), retryStrategy);

public static EntityEntry Refresh(this EntityEntry tracking, RefreshConflict refreshMode)

switch (refreshMode)

case RefreshConflict.StoreWins:

// When entity is already deleted in database, Reload sets tracking state to Detached.

// When entity is already updated in database, Reload sets tracking state to Unchanged.

tracking.Reload(); // Execute SELECT.

// Hereafter, SaveChanges ignores this entity.

break;

case RefreshConflict.ClientWins:

PropertyValues databaseValues = tracking.GetDatabaseValues(); // Execute SELECT.

if (databaseValues == null)

// When entity is already deleted in database, there is nothing for client to win against.

// Manually set tracking state to Detached.

tracking.State = EntityState.Detached;

// Hereafter, SaveChanges ignores this entity.

else

// When entity is already updated in database, refresh original values, which go to in WHERE clause.

tracking.OriginalValues.SetValues(databaseValues);

// Hereafter, SaveChanges executes UPDATE/DELETE for this entity, with refreshed values in WHERE clause.

break;

case RefreshConflict.MergeClientAndStore:

PropertyValues databaseValues = tracking.GetDatabaseValues(); // Execute SELECT.

if (databaseValues == null)

// When entity is already deleted in database, there is nothing for client to merge with.

// Manually set tracking state to Detached.

tracking.State = EntityState.Detached;

// Hereafter, SaveChanges ignores this entity.

else

// When entity is already updated, refresh original values, which go to WHERE clause.

PropertyValues originalValues = tracking.OriginalValues.Clone();

tracking.OriginalValues.SetValues(databaseValues);

// If database has an different value for a property, then retain the database value.

databaseValues.Properties // Navigation properties are not included.

.Where(property => !object.Equals(originalValues[property.Name], databaseValues[property.Name]))

.ForEach(property => tracking.Property(property.Name).IsModified = false);

// Hereafter, SaveChanges executes UPDATE/DELETE for this entity, with refreshed values in WHERE clause.

break;

return tracking;

}

internal static void SaveChanges(AdventureWorks adventureWorks1, AdventureWorks adventureWorks2)

int id = 950;

Product productCopy1 = adventureWorks1.Products.Find(id);

Product productCopy2 = adventureWorks2.Products.Find(id);

productCopy1.Name = nameof(adventureWorks1);

productCopy1.ListPrice = 100;

adventureWorks1.SaveChanges();

productCopy2.Name = nameof(adventureWorks2);

productCopy2.ProductSubcategoryID = 1;

adventureWorks2.SaveChanges(RefreshConflict.MergeClientAndStore);

}

Entity Framework Core and LINQ to Entities in Depth (6) Query Data Loading

Fri, 11 Oct 2019 00:00:00 GMT

[LINQ via C# series]

[Entity Framework Core (EF Core) series]

[Entity Framework (EF) series]

After translated to SQL, in LINQ to Entities, sequence queries returning IQueryable<T> implements deferred execution too.

Deferred execution

As previous part discussed, when defining a LINQ to Entities query represented by IQueryable<T>, an expression tree is built, there is no query execution. The execution is deferred until trying to pull the results from the query.

Iterator pattern

IQueryable<T> implements IEnumerable<T>, so values can be pulled from IQueryable<T> with the standard iterator pattern. When trying to pull the first value, EF Core translates LINQ to Entities query to SQL, and execute SQL in the database. The implementation can be demonstrated with the Iterator<T> type from the LINQ to Objects chapter:

public static IEnumerator<TEntity> GetEntityIterator<TEntity>(

this IQueryable<TEntity> query, DbContext dbContext) where TEntity : class

"| |_Compile LINQ expression tree to database expression tree.".WriteLine();

(SelectExpression DatabaseExpression, IReadOnlyDictionary<string, object> Parameters) compilation = dbContext.Compile(query.Expression);

IEnumerator<TEntity> entityIterator = null;

return new Iterator<TEntity>(

start: () =>

"| |_Generate SQL from database expression tree.".WriteLine();

IRelationalCommand sql = dbContext.Generate(compilation.DatabaseExpression);

IEnumerable<TEntity> sqlQuery = dbContext.Set<TEntity>().FromRawSql(

sql: sql.CommandText,

parameters: compilation.Parameters

.Select(parameter => new SqlParameter(parameter.Key, parameter.Value)).ToArray());

entityIterator = sqlQuery.GetEnumerator();

"| |_Execute generated SQL.".WriteLine();

},

moveNext: () => entityIterator.MoveNext(),

getCurrent: () =>

$"| |_Materialize data row to {typeof(TEntity).Name} entity.".WriteLine();

return entityIterator.Current;

},

dispose: () => entityIterator.Dispose(),

end: () => " |_End.".WriteLine()).Start();

}

The following example executes Where and Take query to load 3 products with more than 10 characters in name. It demonstrates how to pull the results from IQueryable<T> with the iterator pattern:

internal static void DeferredExecution(AdventureWorks adventureWorks)

IQueryable<Product> categories = adventureWorks.Products

.Where(product => product.Name.Length > 100)

.Take(3);

"Iterator - Create from LINQ to Entities query.".WriteLine();

using (IEnumerator<Product> iterator = categories.GetEntityIterator(adventureWorks)) // Compile query.

int index = 0;

while (new Func<bool>(() =>

bool moveNext = iterator.MoveNext();

$"|_Iterator - [{index++}] {nameof(IEnumerator<Product>.MoveNext)}: {moveNext}.".WriteLine();

return moveNext; // Generate SQL when first time called.

})())

Product product = iterator.Current;

$"| |_Iterator - [{index}] {nameof(IEnumerator<Product>.Current)}: {product.Name}.".WriteLine();

// Iterator - Create from LINQ to Entities query.

// | |_Compile LINQ expression tree to database expression tree.

// |_Iterator - [0] MoveNext: True.

// | |_Generate SQL from database expression tree.

// | |_Execute generated SQL.

// | |_Materialize data row to Product entity.

// | |_Iterator - [0] Current: ML Crankset.

// |_Iterator - [1] MoveNext: True.

// | |_Materialize data row to Product entity.

// | |_Iterator - [1] Current: HL Crankset.

// |_Iterator - [2] MoveNext: True.

// | |_Materialize data row to Product entity.

// | |_Iterator - [2] Current: Touring-2000 Blue, 60.

// |_Iterator - [3] MoveNext: False.

// |_End.

}

Here for demonstration purpose, the GetEntityIterator extension method of IQueryable<T> is called instead of GetEnumerator. In EF Core, when the iterator is created from IQueryable<T>, the LINQ query expression tree is compiled to database query expression tree. Later, when the iterator’s MoveNext method is called for the first time, the SQL query is generated and executed. In each iteration, an entity is materialized from the SQL execution result.

Lazy evaluation vs. eager evaluation

Deferred execution can be either lazy evaluation or eager evaluation. Internally, EF Core call ADP.NET APIs to execute query, including DbDataReader, etc. DbDataReader is abstract class. EF Core SQL database provider actually uses SqlDataReader in ADO.NET, which is derived from DbDataReader, to load the database query results. By default, when SqlDataReader starts to read data, it streams a number of rows to local buffer through TDS (tabular data stream) protocol. So by default, LINQ to Entities’ deferred execution is neither eager (load all rows when pulling the first result), nor totally lazy (load 1 result when pulling each result).

When retry logic is specified for connection resiliency, EF Core become eager evaluation. When trying to pull the first query result, EF Core call DbDataReader to load all results from database.

Explicit loading

After an entity is queried, its related entities can be loaded through the navigation property. DbContext.Entry method accepts an entity of type TEntity, and returns Microsoft.EntityFrameworkCore.ChangeTracking.EntityEntry<TEntity>, which represents that entity’s tracking and loading information. EntityEntry<TEntity> provides a Reference method to return Microsoft.EntityFrameworkCore.ChangeTracking.ReferenceEntry<TEntity, TProperty> instance, which represents the tracking and loading information of a single related entity from reference navigation property. EntityEntry<TEntity> also provides a Collection method to return Microsoft.EntityFrameworkCore.ChangeTracking.ReferenceEntry.CollectionEntry<TEntity, TProperty>, which represents the tracking and loading information of multiple related entities from collection navigation property. These related entities in the navigation properties can be manually loaded by calling ReferenceEntry<TEntity, TProperty>.Load and CollectionEntry<TEntity, TProperty>.Load:

internal static void ExplicitLoading(AdventureWorks adventureWorks)

ProductSubcategory subcategory = adventureWorks.ProductSubcategories.First(); // Execute query.

// SELECT TOP(1) [p].[ProductSubcategoryID], [p].[Name], [p].[ProductCategoryID]

// FROM [Production].[ProductSubcategory] AS [p]

subcategory.Name.WriteLine();

adventureWorks

.Entry(subcategory) // Return EntityEntry<ProductSubcategory>.

.Reference(entity => entity.ProductCategory) // Return ReferenceEntry<ProductSubcategory, ProductCategory>.

.Load(); // Execute query.

// exec sp_executesql N'SELECT [e].[ProductCategoryID], [e].[Name]

// FROM [Production].[ProductCategory] AS [e]

// WHERE [e].[ProductCategoryID] = @__get_Item_0',N'@__get_Item_0 int',@__get_Item_0=1

subcategory.ProductCategory.Name.WriteLine();

adventureWorks

.Entry(subcategory) // Return EntityEntry<ProductSubcategory>.

.Collection(entity => entity.Products) // Return CollectionEntry<ProductSubcategory, Product>.

.Load(); // Execute query.

// exec sp_executesql N'SELECT [e].[ProductID], [e].[ListPrice], [e].[Name], [e].[ProductSubcategoryID]

// FROM [Production].[Product] AS [e]

// WHERE [e].[ProductSubcategoryID] = @__get_Item_0',N'@__get_Item_0 int',@__get_Item_0=1

subcategory.Products.WriteLines(product => product.Name);

}

When the Load method is called, the related entities are queried, and become available through the navigation properties. Besides loading the full entities, explicit lazy loading also support custom query. The following example uses the reference navigation property and collection navigation property as LINQ to Entities data sources, by calling ReferenceEntry<TEntity, TProperty>.Query and CollectionEntry<TEntity, TProperty>.Query:

internal static void ExplicitLoadingWithQuery(AdventureWorks adventureWorks)

ProductSubcategory subcategory = adventureWorks.ProductSubcategories.First(); // Execute query.

// SELECT TOP(1) [p].[ProductSubcategoryID], [p].[Name], [p].[ProductCategoryID]

// FROM [Production].[ProductSubcategory] AS [p]

subcategory.Name.WriteLine();

string categoryName = adventureWorks

.Entry(subcategory).Reference(entity => entity.ProductCategory)

.Query() // Return IQueryable<ProductCategory>.

.Select(category => category.Name).Single(); // Execute query.

// exec sp_executesql N'SELECT TOP(2) [e].[Name]

// FROM [Production].[ProductCategory] AS [e]

// WHERE [e].[ProductCategoryID] = @__get_Item_0',N'@__get_Item_0 int',@__get_Item_0=1

categoryName.WriteLine();

IQueryable<string>products = adventureWorks

.Entry(subcategory).Collection(entity => entity.Products)

.Query() // Return IQueryable<Product>.

.Select(product => product.Name); // Execute query.

// exec sp_executesql N'SELECT [e].[Name]

// FROM [Production].[Product] AS [e]

// WHERE [e].[ProductSubcategoryID] = @__get_Item_0',N'@__get_Item_0 int',@__get_Item_0=1

products.WriteLines();

}

Eager loading

In explicit loading, after an entity is queried, its related entities are loaded separately. In eager loading, when an entity is queried, its related entities are loaded during the same query. To enable eager loading, call Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions’ Include method, which is an extension method for IQueryable<T>:

internal static void EagerLoadingWithInclude(AdventureWorks adventureWorks)

IQueryable<ProductSubcategory> subcategoriesWithCategory = adventureWorks.ProductSubcategories

.Include(subcategory => subcategory.ProductCategory);

subcategoriesWithCategory.WriteLines(subcategory =>

$"{subcategory.ProductCategory.Name}: {subcategory.Name}");

// SELECT [subcategory].[ProductSubcategoryID], [subcategory].[Name], [subcategory].[ProductCategoryID], [p].[ProductCategoryID], [p].[Name]

// FROM [Production].[ProductSubcategory] AS [subcategory]

// INNER JOIN [Production].[ProductCategory] AS [p] ON [subcategory].[ProductCategoryID] = [p].[ProductCategoryID]

IQueryable<ProductSubcategory> subcategoriesWithProducts = adventureWorks.ProductSubcategories

.Include(subcategory => subcategory.Products);

subcategoriesWithProducts.WriteLines(subcategory => $@"{subcategory.Name}: {string.Join(

", ", subcategory.Products.Select(product => product.Name))}");

// SELECT [subcategory].[ProductSubcategoryID], [subcategory].[Name], [subcategory].[ProductCategoryID]

// FROM [Production].[ProductSubcategory] AS [subcategory]

// ORDER BY [subcategory].[ProductSubcategoryID]

// SELECT [p].[ProductID], [p].[ListPrice], [p].[Name], [p].[ProductSubcategoryID], [p].[RowVersion]

// FROM [Production].[Product] AS [p]

// WHERE EXISTS (

// SELECT 1

// FROM [Production].[ProductSubcategory] AS [subcategory]

// WHERE [p].[ProductSubcategoryID] = [subcategory].[ProductSubcategoryID])

// ORDER BY [p].[ProductSubcategoryID]

}

Eager loading related entity through reference navigation property is translated to INNER JOIN. Eager loading through collection navigation property is translated to 2 SQL queries for 2 types of entities. More queries can be chained after calling Include.

In EF Core, ThenInclude can be called for eager loading of multiple levels of related entities:

internal static void EagerLoadingMultipleLevels(AdventureWorks adventureWorks)

IQueryable<Product>products = adventureWorks.Products

.Include(product => product.ProductProductPhotos)

.ThenInclude(productProductPhoto => productProductPhoto.ProductPhoto);

products.WriteLines(product => $@"{product.Name}: {string.Join(

", ",

product.ProductProductPhotos.Select(productProductPhoto =>

productProductPhoto.ProductPhoto.LargePhotoFileName))}");

// SELECT [product].[ProductID], [product].[ListPrice], [product].[Name], [product].[ProductSubcategoryID], [product].[RowVersion]

// FROM [Production].[Product] AS [product]

// ORDER BY [product].[ProductID]

// SELECT [p].[ProductID], [p].[ProductPhotoID], [p0].[ProductPhotoID], [p0].[LargePhotoFileName], [p0].[ModifiedDate]

// FROM [Production].[ProductProductPhoto] AS [p]

// INNER JOIN [Production].[ProductPhoto] AS [p0] ON [p].[ProductPhotoID] = [p0].[ProductPhotoID]

// WHERE EXISTS (

// SELECT 1

// FROM [Production].[Product] AS [product]

// WHERE [p].[ProductID] = [product].[ProductID])

// ORDER BY [p].[ProductID]

}

Lazy loading

EF Core also supports lazy loading.

public partial class AdventureWorks

public AdventureWorks(DbConnection connection = null, bool lazyLoading = true)

: base(GetDbContextOptions(connection, lazyLoading))

private static DbContextOptions GetDbContextOptions(

DbConnection connection = null, bool lazyLoading = true) =>

new DbContextOptionsBuilder<AdventureWorks>()

.UseLazyLoadingProxies(lazyLoading)

.UseSqlServer(

connection: connection ??

new SqlConnection(ConnectionStrings.AdventureWorks),

sqlServerOptionsAction: options => options.EnableRetryOnFailure(

maxRetryCount: 5, maxRetryDelay: TimeSpan.FromSeconds(30),

errorNumbersToAdd: null))

.Options;

}

When an entity’s navigation property is accessed, the related entities are queried and loaded automatically:

internal static void LazyLoading(AdventureWorks adventureWorks)

ProductSubcategory subcategory = adventureWorks.ProductSubcategories.First(); // Execute query.

// SELECT TOP(1) [p].[ProductSubcategoryID], [p].[Name], [p].[ProductCategoryID]

// FROM [Production].[ProductSubcategory] AS [p]

subcategory.Name.WriteLine();

ProductCategory category = subcategory.ProductCategory; // Execute query.

// exec sp_executesql N'SELECT [e].[ProductCategoryID], [e].[Name]

// FROM [Production].[ProductCategory] AS [e]

// WHERE [e].[ProductCategoryID] = @__get_Item_0',N'@__get_Item_0 int',@__get_Item_0=1

category.Name.WriteLine();

ICollection<Product> products = subcategory.Products; // Execute query.

// exec sp_executesql N'SELECT [e].[ProductID], [e].[ListPrice], [e].[Name], [e].[ProductSubcategoryID], [e].[RowVersion]

// FROM [Production].[Product] AS [e]

// WHERE [e].[ProductSubcategoryID] = @__get_Item_0',N'@__get_Item_0 int',@__get_Item_0=1

products.WriteLines(product => product.Name);

}

The N + 1 problem

Sometimes lazy loading can cause the “N + 1 queries” problem. The following example queries the subcategories, and pulls each subcategory’s information:

internal static void MultipleLazyLoading(AdventureWorks adventureWorks)

ProductSubcategory[] subcategories = adventureWorks.ProductSubcategories.ToArray(); // Execute query.

// SELECT [p].[ProductSubcategoryID], [p].[Name], [p].[ProductCategoryID]

// FROM [Production].[ProductSubcategory] AS [p]

subcategories.WriteLines(subcategory =>

$"{subcategory.Name} ({subcategory.ProductCategory.Name})"); // Execute query.

// exec sp_executesql N'SELECT [e].[ProductCategoryID], [e].[Name]

// FROM [Production].[ProductCategory] AS [e]

// WHERE [e].[ProductCategoryID] = @__get_Item_0',N'@__get_Item_0 int',@__get_Item_0=1

// exec sp_executesql N'SELECT [e].[ProductCategoryID], [e].[Name]

// FROM [Production].[ProductCategory] AS [e]

// WHERE [e].[ProductCategoryID] = @__get_Item_0',N'@__get_Item_0 int',@__get_Item_0=2

// ...

}

When loading the subcategories, 1 database query is executed. When each subcategory’s related category is pulled through the navigation property, it is loaded instantly, if not loaded yet. So in total there are N queries for related categories + 1 query for subcategories executed. For better performance in this kind of scenario, eager loading or inner join should be used to load all entities and related entities with 1 single query.

Disable lazy loading

There are some scenarios where lazy loading needs to be disabled, like entity serialization. There are several ways to disable lazy loading for different scopes

· To globally disable lazy loading for specific navigation properties, just do not mark it as virtual, so that the derived proxy entity cannot override it with the lazy load implementation.

· To disable lazy loading for specific DbContext or specific query, call DbContext.Configuration to get a DbConfiguration instance, and set its LazyLoadingEnabled property to false.

internal static void DisableLazyLoading()

using (AdventureWorks adventureWorks = new AdventureWorks(lazyLoading: false))

ProductSubcategory subcategory = adventureWorks.ProductSubcategories.First(); // Execute query.

subcategory.Name.WriteLine();

ProductCategory category = subcategory.ProductCategory; // No query.

(category == null).WriteLine(); // True

ICollection<Product> products = subcategory.Products; // No query.

(products == null).WriteLine(); // True

}

Entity Framework Core and LINQ to Entities in Depth (5) Query Translation Implementation

Thu, 10 Oct 2019 00:00:00 GMT

[LINQ via C# series]

[Entity Framework Core (EF Core) series]

[Entity Framework (EF) series]

Regarding different database systems can have different query languages or different query APIs, EF Core implement a provider model to work with different kinds of databases. In EF Core, the base libraries are the Microsoft.EntityFrameworkCore and Microsoft.EntityFrameworkCore.Relational NuGet packages. Microsoft.EntityFrameworkCore provides the database provider contracts as Microsoft.EntityFrameworkCore.Storage.IDatabaseProviderServices interface. And the SQL database support is implemented by the Microsoft.EntityFrameworkCore,SqlServer NuGet package, which provides Microsoft.EntityFrameworkCore.Storage.Internal.SqlServerDatabaseProviderServices type to implement IDatabaseProviderServices. There are other libraries for different databases, like Microsoft.EntityFrameworkCore.SQLite NuGet package for SQLite, etc.

With this provider model, EF Core breaks the translation into 2 parts. First, IQueryable<T> queries work with expression trees, and EF Core base libraries translate these .NET expression tree to generic, intermediate database expression tree; Then the specific EF Core database provider is responsible to generate query language for the specific database.

Code to LINQ expression tree

Before translation, .NET expression tree must be built to represent the query logic. As fore mentioned, expression tree enables function as data. In C#, an expression tree shares the same lambda expression syntax as anonymous functions, but is compiled to abstract syntactic tree representing function’s source code. In LINQ, IQueryable<T> utilizes expression tree to represent the abstract syntactic structure of a remote query.

IQueryable<T> and IQueryProvider

IQueryable<T> has been demonstrated:

namespace System.Linq

public interface IQueryable<out T> : IEnumerable<T>, IEnumerable, IQueryable

// IEnumerator<T> GetEnumerator(); from IEnumerable<T>.

// Type ElementType { get; } from IQueryable.

// Expression Expression { get; } from IQueryable.

// IQueryProvider Provider { get; } from IQueryable.

}

It is a wrapper of iterator factory, an element type, an expression tree representing the current query’s logic, and a query provider of IQueryProvider type:

namespace System.Linq

public interface IQueryProvider

IQueryable CreateQuery(Expression expression);

IQueryable<TElement> CreateQuery<TElement>(Expression expression);

object Execute(Expression expression);

TResult Execute<TResult>(Expression expression);

}

IQueryProvider has CreateQuery and Execute methods, all accepting a expression tree parameter. CreateQuery returns an IQueryable<T> query, and Execute returns a query result. These methods are called by the standard queries internally.

Standard remote queries

Queryable provides 2 kinds of queries, sequence queries returning IQueryable<T> query, and value queries returning a query result. Take Where, Select, and First as examples, the following are their implementations:

namespace System.Linq

public static class Queryable

public static IQueryable<TSource> Where<TSource>(

this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate)

Func<IQueryable<TSource>, Expression<Func<TSource, bool>>, IQueryable<TSource>> currentMethod = Where;

MethodCallExpression whereCallExpression = Expression.Call(

method: currentMethod.Method,

arg0: source.Expression,

arg1: Expression.Quote(predicate));

return source.Provider.CreateQuery<TSource>(whereCallExpression);

public static IQueryable<TResult> Select<TSource, TResult>(

this IQueryable<TSource>source, Expression<Func<TSource, TResult>> selector)

Func<IQueryable<TSource>, Expression<Func<TSource, TResult>>, IQueryable<TResult>> currentMethod =

Select;

MethodCallExpression selectCallExpression = Expression.Call(

method: currentMethod.Method,

arg0: source.Expression,

arg1: Expression.Quote(selector));

return source.Provider.CreateQuery<TResult>(selectCallExpression);

public static TSource First<TSource>(

this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate)

Func<IQueryable<TSource>, Expression<Func<TSource, bool>>, TSource>currentMethod = First;

MethodCallExpression firstCallExpression = Expression.Call(

method: currentMethod.Method,

arg0: source.Expression,

arg1: Expression.Quote(predicate));

return source.Provider.Execute<TSource>(firstCallExpression);

public static TSource First<TSource>(this IQueryable<TSource>source)

Func<IQueryable<TSource>, TSource>currentMethod = First;

MethodCallExpression firstCallExpression = Expression.Call(

method: currentMethod.Method,

arg0: source.Expression);

return source.Provider.Execute<TSource>(firstCallExpression);

// Other members.

}

They just build a MethodCallExpression expression, representing the current query is called. Then they obtain query provider from source’s Provider property. The sequence queries call query provider’s CreateQuery method to return IQueryable<T> query, and the value queries call query provider’s Execute method to return a query result. All standard queries are implemented in this pattern except AsQueryable.

Build LINQ to Entities abstract syntax tree

With above Where and Select queries, a simple LINQ to Entities query can be implemented to return an IQueryable<T> of values:

internal static void WhereAndSelect(AdventureWorks adventureWorks)

// IQueryable<string> products = adventureWorks.Products

// .Where(product => product.Name.Length> 10)

// .Select(product => product.Name);

IQueryable<Product> sourceQueryable = adventureWorks.Products;

IQueryable<Product> whereQueryable = sourceQueryable.Where(product => product.Name.Length > 10);

IQueryable<string> selectQueryable = whereQueryable.Select(product => product.Name); // Define query.

foreach (string result in selectQueryable) // Execute query.

result.WriteLine();

}

The above example filters the products with Name longer than 10 characters, and queries the products’ Names. By desugaring the lambda expressions, and unwrapping the standard queries, the above LINQ to Entities query is equivalent to:

internal static void WhereAndSelectLinqExpressions(AdventureWorks adventureWorks)

IQueryable<Product>sourceQueryable = adventureWorks.Products; // DbSet<Product>.

ConstantExpression sourceConstantExpression = (ConstantExpression)sourceQueryable.Expression;

IQueryProvider sourceQueryProvider = sourceQueryable.Provider; // EntityQueryProvider.

// Expression<Func<Product, bool>> predicateExpression = product => product.Name.Length > 10;

ParameterExpression productParameterExpression = Expression.Parameter(typeof(Product), "product");

Expression<Func<Product, bool>>predicateExpression = Expression.Lambda<Func<Product, bool>>(

body: Expression.GreaterThan(

left: Expression.Property(

expression: Expression.Property(

expression: productParameterExpression, propertyName: nameof(Product.Name)),

propertyName: nameof(string.Length)),

right: Expression.Constant(10)),

parameters: productParameterExpression);

// IQueryable<Product> whereQueryable = sourceQueryable.Where(predicateExpression);

Func<IQueryable<Product>, Expression<Func<Product, bool>>, IQueryable<Product>>whereMethod = Queryable.Where;

MethodCallExpression whereCallExpression = Expression.Call(

method: whereMethod.Method,

arg0: sourceConstantExpression,

arg1: Expression.Quote(predicateExpression));

IQueryable<Product>whereQueryable = sourceQueryProvider

.CreateQuery<Product>(whereCallExpression); // EntityQueryable<Product>.

IQueryProvider whereQueryProvider = whereQueryable.Provider; // EntityQueryProvider.

// Expression<Func<Product, string>> selectorExpression = product => product.Name;

Expression<Func<Product, string>> selectorExpression = Expression.Lambda<Func<Product, string>>(

body: Expression.Property(productParameterExpression, nameof(Product.Name)),

parameters: productParameterExpression);

// IQueryable<string> selectQueryable = whereQueryable.Select(selectorExpression);

Func<IQueryable<Product>, Expression<Func<Product, string>>, IQueryable<string>>selectMethod = Queryable.Select;

MethodCallExpression selectCallExpression = Expression.Call(

method: selectMethod.Method,

arg0: whereCallExpression,

arg1: Expression.Quote(selectorExpression));

IQueryable<string>selectQueryable = whereQueryProvider

.CreateQuery<string>(selectCallExpression); // EntityQueryable<Product>.

using (IEnumerator<string>iterator = selectQueryable.GetEnumerator()) // Execute query.

while (iterator.MoveNext())

iterator.Current.WriteLine();

}

Here are the steps how the fluent query builds its query expression tree:

· Build data source:

o The initial source IQueryable<T> is a DbSet<T> instance given by EF Core. It wraps an expression and a query provider:

§ The expression is a ConstantExpression expression representing the data source.

§ The query provider is an EntityQueryProvider instance automatically created by EF Core.

· Build Where query:

o A predicate expression is built for Where,

o Where accepts the IQueryable<T> source. But actually Where only needs the source’s expression and query provider. A MethodCallExpression expression is built to represent a call of Where itself with 2 arguments, the source and the predicate expression. Then source query provider’s CreateQuery method is called with the MethodCallExpression expression just built, and return an IQueryable<T> query, which wraps:

§ The MethodCallExpression expression representing current Where call

§ The same query privider from its source.

· Build Select query:

o A selector expression is built for Select

o Select accepts the IQueryable<T> returned by Where as source. Again, Select only needs the expression and query provider from source. A MethodCallExpression expression is built to represent a call to Select itself with 2 arguments, the source and the selector expression. Then source query provider’s CreateQuery method is called with the MethodCallExpression expression just built, and return an IQueryable<T> query, which wraps:

§ The MethodCallExpression expression representing current Select call

§ The same query privider from its source.

So, the final IQueryable<T> query’s Expression property is the final abstract syntactic tree, which represents the entire LINQ to Entities query logic:

MethodCallExpression (NodeType = Call, Type = IQueryable<string>)

|_Method = Queryable.Select<Product, string>

|_Object = null

|_Arguments

|_MethodCallExpression (NodeType = Call, Type = IQueryable<Product>)

| |_Method = Queryable.Where<Product>

| |_Object = null

| |_Arguments

| |_ConstantExpression (NodeType = Constant, Type = IQueryable<Product>)

| | |_Value = new EntityQueryable<Product>(adventureWorks.GetService<IAsyncQueryProvider>())

| |_UnaryExpression (NodeType = Quote, Type = Expression<Func<Product, bool>>)

| |_Operand

| |_Expression<Func<Product, bool>> (NodeType = Lambda, Type = Func<Product, bool>)

| |_Parameters

| | |_ParameterExpression (NodeType = Parameter, Type = Product)

| | |_Name = "product"

| |_Body

| |_BinaryExpression (NodeType = GreaterThan, Type = bool)

| |_Left

| | |_MemberExpression (NodeType = MemberAccess, Type = int)

| | |_Member = "Length"

| | |_Expression

| | |_MemberExpression (NodeType = MemberAccess, Type = string)

| | |_Member = "Name"

| | |_Expression

| | |_ParameterExpression (NodeType = Parameter, Type = Product)

| | |_Name = "product"

| |_Right

| |_ConstantExpression (NodeType = Constant, Type = int)

| |_Value = 10

|_UnaryExpression (NodeType = Quote, Type = Expression<Func<Product, string>>)

|_Operand

|_Expression<Func<Product, string>> (NodeType = Lambda, Type = Func<Product, string>)

|_Parameters

| |_ParameterExpression (NodeType = Parameter, Type = Product)

| |_Name = "product"

|_Body

|_MemberExpression (NodeType = MemberAccess, Type = string)

|_Member = "Name"

|_Expression

|_ParameterExpression (NodeType = Parameter, Type = Product)

|_Name = "product"

This also demonstrates that lambda expression, extension methods, and LINQ query expression are powerful language features of C#. The above a rich abstract syntactic tree can be built by C# code as simple as:

internal static void WhereAndSelectQuery(AdventureWorks adventureWorks)

IQueryable<string>products = adventureWorks.Products

.Where(product => product.Name.Length > 10)

.Select(product => product.Name);

// Equivalent to:

// IQueryable<string> products =

// from product in adventureWorks.Products

// where product.Name.Length > 10

// select product.Name;

}

The other kind of query returning a single value works in the similar way. Take above First as example:

internal static void SelectAndFirst(AdventureWorks adventureWorks)

// string first = adventureWorks.Products.Select(product => product.Name).First();

IQueryable<Product> sourceQueryable = adventureWorks.Products;

IQueryable<string>selectQueryable = sourceQueryable.Select(product => product.Name);

string first = selectQueryable.First().WriteLine(); // Execute query.

}

Here the initial source and and Select query are the same as the previous example. So this time, just unwrap the First query. The above First query is equivalent to:

internal static void SelectAndFirstLinqExpressions(AdventureWorks adventureWorks)

IQueryable<Product>sourceQueryable = adventureWorks.Products;

IQueryable<string>selectQueryable = sourceQueryable.Select(product => product.Name);

MethodCallExpression selectCallExpression = (MethodCallExpression)selectQueryable.Expression;

IQueryProvider selectQueryProvider = selectQueryable.Provider; // DbQueryProvider.

// string first = selectQueryable.First();

Func<IQueryable<string>, string> firstMethod = Queryable.First;

MethodCallExpression firstCallExpression = Expression.Call(

method: firstMethod.Method, arg0: selectCallExpression);

string first = selectQueryProvider.Execute<string>(firstCallExpression).WriteLine(); // Execute query.

}

In First query, the MethodCallExpression expression is built to represent current First call. The difference is, then query provider’s Execute method is called instead of CreateQuery, so that a query result is returned instead of a query.

Similarly, the last expression tree built inside First, is the final abstract syntactic tree, which represents the entire LINQ to Entities query logic:

MethodCallExpression (NodeType = Call, Type = string)

|_Method = Queryable.First<string>

|_Object = null

|_Arguments

|_MethodCallExpression (NodeType = Call, Type = IQueryable<string>)

|_Method = Queryable.Select<Product, string>

|_Object = null

|_Arguments

|_ConstantExpression (NodeType = Constant, Type = IQueryable<Product>)

| |_Value = new EntityQueryable<Product>(adventureWorks.GetService<IAsyncQueryProvider>())

|_UnaryExpression (NodeType = Quote, Type = Expression<Func<Product, string>>)

|_Operand

|_Expression<Func<Product, string>> (NodeType = Lambda, Type = Func<Product, string>)

|_Parameters

| |_ParameterExpression (NodeType = Parameter, Type = Product)

| |_Name = "product"

|_Body

|_MemberExpression (NodeType = MemberAccess, Type = string)

|_Member = "Name"

|_Expression

|_ParameterExpression (NodeType = Parameter, Type = Product)

|_Name = "product"

And again, the entire abstract syntactic tree can be built by C# code as simple as:

internal static void SelectAndFirstQuery(AdventureWorks adventureWorks)

string first = adventureWorks.Products.Select(product => product.Name).First();

// Equivalent to:

// string first = (from product in adventureWorks.Products select product.Name).First();

}

.NET expression tree to database expression tree

When LINQ to Entities queries are executed by either pulling values from IQueryable<T>, or calling IQueryProvider.Execute, EF Core compiles .NET expression tree to database expression tree.

Database query abstract syntax tree

The logic of LINQ to Entities can be represented by .NET expression tree, and EF Core also use expression tree to represent the database query logic. For example, EF Core base libraries provides the Microsoft.EntityFrameworkCore.Query.Expressions.SelectExpression type to represent a database SELECT query:

namespace Microsoft.EntityFrameworkCore.Query.Expressions

public class SelectExpression : TableExpressionBase

public virtual IReadOnlyList<Expression> Projection { get; } // SELECT.

public virtual bool IsDistinct { get; set; } // DISTINCT.

public virtual Expression Limit { get; set; } // TOP.

public virtual IReadOnlyList<TableExpressionBase> Tables { get; } // FROM.

public virtual Expression Predicate { get; set; } // WHERE.

public virtual IReadOnlyList<Ordering> OrderBy { get; } // ORDER BY.

public virtual Expression Offset { get; set; } // OFFSET.

public override Type Type { get; }

// Other members.

}

The following are are all the database expressions provided by EF Core and the Remotion.Linq library used by EF Core, which are all derived from the Expression type:

· AggregateExpression

o MaxExpression

o MinExpression

o SumExpression

· AliasExpression

· ColumnExpression

· CountExpression

· DatePartExpression

· DiscriminatorPredicateExpression

· ExistsExpression

· ExplicitCastExpression

· InExpression

· IsNullExpression

· LikeExpression

· NotNullableExpression

· NullConditionalExpression

· PartialEvaluationExceptionExpression

· PropertyParameterExpression

· QuerySourceReferenceExpression

· RowNumberExpression

· SqlFunctionExpression

· StringCompareExpression

· SubQueryExpression

· TableExpressionBase

o CrossJoinExpression

o FromSqlExpression

o JoinExpressionBase

§ InnerJoinExpression

§ LeftOuterJoinExpression

o LateralJoinExpression

o SelectExpression

o TableExpression

· VBStringComparisonExpression

Compile LINQ expressions to database expressions

EF Core calls the third party library Remotion.Linq to compile LINQ expression tree to a query model, then EF Core compiles the query model to database expression tree, which is a SelectExpression instance. The following Compile function demonstrates how the compilation can be done. It accepts a LINQ expression tree, and returns a tuple of SelectExpression, and its parameters if any:

public static (SelectExpression, IReadOnlyDictionary<string, object>) Compile(this DbContext dbContext, Expression linqExpression)

QueryContext queryContext = dbContext

.GetService<IQueryContextFactory>()

.Create();

QueryCompilationContext compilationContext = dbContext

.GetService<IQueryCompilationContextFactory>()

.Create(async: false);

QueryCompiler queryCompiler = (QueryCompiler)dbContext.GetService<IQueryCompiler>();

linqExpression = queryCompiler.ExtractParameters(

linqExpression,

queryContext,

dbContext.GetService<IDiagnosticsLogger<DbLoggerCategory.Query>>());

linqExpression = dbContext

.GetService<IQueryTranslationPreprocessorFactory>()

.Create(compilationContext)

.Process(linqExpression);

ShapedQueryExpression queryExpression = (ShapedQueryExpression)dbContext

.GetService<IQueryableMethodTranslatingExpressionVisitorFactory>()

.Create(dbContext.Model)

.Visit(linqExpression);

queryExpression = (ShapedQueryExpression)dbContext

.GetService<IQueryTranslationPostprocessorFactory>()

.Create(compilationContext)

.Process(queryExpression);

return ((SelectExpression)queryExpression.QueryExpression, queryContext.ParameterValues);

}

So above Where and Select query’s expression tree can be compiled as:

internal static void CompileWhereAndSelectExpressions(AdventureWorks adventureWorks)

Expression linqExpression =adventureWorks.Products

.Where(product => product.Name.Length > 10)

.Select(product => product.Name).Expression;

(SelectExpression DatabaseExpression, IReadOnlyDictionary<string, object> Parameters) compilation =

adventureWorks.Compile(linqExpression);

compilation.DatabaseExpression.WriteLine();

compilation.Parameters.WriteLines(parameter => $"{parameter.Key}: {parameter.Value}");

}

The compiled SelectExpression is the same as the following SelectExpression built on the fly:

internal static SelectExpression WhereAndSelectDatabaseExpressions(AdventureWorks adventureWorks)

IQueryableMethodTranslatingExpressionVisitorFactory expressionVisitorFactory = adventureWorks

.GetService<IQueryableMethodTranslatingExpressionVisitorFactory>();

ISqlExpressionFactory expressionFactory = adventureWorks.GetService<ISqlExpressionFactory>();

IEntityType entityType = adventureWorks.Model.FindEntityType(typeof(Product));

SelectExpression databaseExpression = expressionFactory.Select(entityType);

EntityProjectionExpression projectionExpression = (EntityProjectionExpression)databaseExpression.GetMappedProjection(new ProjectionMember());

ColumnExpression columnExpression = projectionExpression.BindProperty(entityType.FindProperty(nameof(Product.Name)));

databaseExpression.ApplyPredicate(expressionFactory.MakeBinary(

ExpressionType.GreaterThan,

expressionFactory.Convert(

expressionFactory.Function("LEN", new SqlExpression[] { columnExpression }, typeof(long)),

typeof(int)),

new SqlConstantExpression(Expression.Constant(10), null),

null));

databaseExpression.AddToProjection(columnExpression);

return databaseExpression.WriteLine();

}

This abstract syntactic tree can be visualized as:

SelectExpression (NodeType = Extension, Type = string)

|_Porjection

| |_ColumnExpression (NodeType = Extension, Type = string)

| |_Name = "Name"

| |_Property = Product.Name

| |_Table

| |_TableExpression (NodeType = Extension, Type = object)

| |_Schema = "Production"

| |_Name = "Product"

| |_Alias = "product"

|_Tables

| |_TableExpression (NodeType = Extension, Type = object)

| |_Schema = "Production"

| |_Name = "Product"

| |_Alias = "product"

|_Predicate

|_BinaryExpression (NodeType = GreaterThan, Type = bool)

|_left

| |_ExplicitCastExpression (NodeType = Extension, Type = int)

| |_Operand

| |_SqlFunctionExpression (NodeType = Extension, Type = int)

| |_FunctionName = "LEN"

| |_Arguments

| |_ColumnExpression (NodeType = Extension, Type = string)

| |_Name = "Name"

| |_Property = Product.Name

| |_Table

| |_TableExpression (NodeType = Extension, Type = object)

| |_Schema = "Production"

| |_Name = "Product"

| |_Alias = "product"

|_Right

|_ConstantExpression (NodeType = Constant, Type = int)

|_Value = 1

Similarly, the other Select and First query’s expression tree is compiled to abstract syntax tree the same as the following:

internal static SelectExpression SelectAndFirstDatabaseExpressions(AdventureWorks adventureWorks)

IQueryableMethodTranslatingExpressionVisitorFactory expressionVisitorFactory = adventureWorks

.GetService<IQueryableMethodTranslatingExpressionVisitorFactory>();

ISqlExpressionFactory expressionFactory = adventureWorks.GetService<ISqlExpressionFactory>();

IEntityType entityType = adventureWorks.Model.FindEntityType(typeof(Product));

SelectExpression databaseExpression = expressionFactory.Select(entityType);

EntityProjectionExpression projectionExpression = (EntityProjectionExpression)databaseExpression.GetMappedProjection(new ProjectionMember());

ColumnExpression columnExpression = projectionExpression.BindProperty(entityType.FindProperty(nameof(Product.Name)));

databaseExpression.AddToProjection(columnExpression);

databaseExpression.ApplyLimit(expressionFactory.ApplyDefaultTypeMapping(new SqlConstantExpression(Expression.Constant(1), null)));

return databaseExpression.WriteLine();

}

And this abstract syntactic tree can be visualized as:

SelectExpression (NodeType = Extension, Type = string)

|_Limit

| |_ConstantExpression (NodeType = Constant, Type = int)

| |_Value = 1

|_Porjection

| |_ColumnExpression (NodeType = Extension, Type = string)

| |_Name = "Name"

| |_Property = Product.Name

| |_Table

| |_TableExpression (NodeType = Extension, Type = object)

| |_Schema = "Production"

| |_Name = "Product"

| |_Alias = "product"

|_Tables

|_TableExpression (NodeType = Extension, Type = object)

|_Schema = "Production"

|_Name = "Product"

|_Alias = "product"

Compile LINQ queries

EF Core first calls Remotion.Linq library to compile LINQ query function call nodes to QueryModel. Under Remotion.Linq.Parsing.Structure.IntermediateModel namespace, Remotion.Linq provides IExpressionNode interface, and many types implementing that interface, where each type can process a certain kind of query function call, for example:

· MethodCallExpression node representing Queryable.Where call is processed by WhereExpressionNode, and converted to Remotion.Linq.Clauses.WhereClause, which is a part of QueryModel

· MethodCallExpression node representing Queryable.Select call is processed by SelectExpressionNode, and converted to Remotion.Linq.Clauses.SelectClause, which is a part of QueryModel

· MethodCallExpression node representing Queryable.First or Queryable.FirstOrDefault call is processed by FirstExpressionNode, and converted to Remotion.Linq.Clauses.ResultOperators.FirstResultOperator, which is a part of QueryModel

etc. Then EF Core continues to compile QueryModel to SelectExpression. For example:

· WhereClause is converted to predicate child nodes of the SelectExpression

· SelectClause is converted to projection child nodes of the SelectExpression

· FirstResultOperator is converted to limit child node of the SelectExpression

etc.

Compile .NET API calls

The above Where query’s predicate has a logic to call string.Length and compare the result to a constant. EF Core provides translator types under Microsoft.EntityFrameworkCore.Query.ExpressionTranslators.Internal namespace to translate these .NET API calls. Here MemberExpression node representing string.Length call is processed by SqlServerStringLengthTranslator, and converted to a SqlFunctionExpression node representing SQL database function LEN call:

namespace Microsoft.EntityFrameworkCore.Query.ExpressionTranslators.Internal

public class SqlServerStringLengthTranslator : IMemberTranslator

public virtual Expression Translate(MemberExpression memberExpression) =>

memberExpression.Expression != null

&&memberExpression.Expression.Type == typeof(string)

&& memberExpression.Member.Name == nameof(string.Length)

? new SqlFunctionExpression("LEN", memberExpression.Type, new Expression[] { memberExpression.Expression })

: null;

}

There are many other translators to cover other basic .NET APIs of System.String, System.Enum, System.DateTime, System.Guid, System.Math, for example:

· MethodCallExpression node representing string.Contains call (e.g. product.Name.Contains(“M”)) is processed by SqlServerContainsOptimizedTranslator, and converted to a BinaryExpression node representing SQL database int comparison, where the left child node is a SqlFunctionExpression node representing SQL database function CHARINDEX call, and the right child node is a ConstantExpression node representing 0 (e.g. CHARINDEX(N'M', product.Name) > 0)

· MethodCallExpression node representing Math.Ceiling call is processed by SqlServerMathCeilingTranslator, and converted to SqlFunctionExpression node representing SQL database function CEILING call

· MemberExpression node representing DateTime.Now or DateTime.UtcNow property access, is processed by SqlServerDateTimeNowTranslator, and converted to SqlFunctionExpression node representing SQL database function GETDATE or GETUTCDATE call

· The extension methods for EF.Functions are also translated to SQL database function calls or operators. For example, EF.Functions.Like is processed by LikeTranslator, and converted to LikeExpression node representing LIKE operator.

etc.

There are also a few other APIs covered with other EF Core components. For example, In Remotion.Linq, MethodCallExpression node representing Enumerable.Contains or List<T>.Contains call is converted to to Remotion.Linq.Clauses.ResultOperators.ContainsResultOperator. Then in EF Core, ContainsResultOperator is processed by Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.SqlTranslatingExpressionVisitor. and converted to InExpression node representing SQL database IN operation.

Remote API call vs. local API call

Apparently EF Core can only compile the supported .NET API calls, like the above string.Length call. It cannot compile arbitrary API calls. The following example wraps the string.Length call and result comparison with constant into a custom predicate:

private static bool FilterName(string name) => name.Length > 10;

internal static void WhereAndSelectWithCustomPredicate(AdventureWorks adventureWorks)

IQueryable<Product>source = adventureWorks.Products;

IQueryable<string>products = source

.Where(product => FilterName(product.Name))

.Select(product => product.Name); // Define query.

products.WriteLines(); // Execute query.

// SELECT [product].[Name]

// FROM [Production].[Product] AS [product]

}

At compile time, the predicate expression tree has a MethodCallExpression node representing FilterName call, which apparently cannot be compiled to SQL by EF Core. In this case, EF Core execute FilterName locally.

Compile database functions and operators

Some database APIs cannot be translated from .NET Standard APIs. For example, there is no mapping .NET API for SQL database LIKE operator, DATEDIFF function, etc. EF Core defines mapping functions to address these scenarios. These functions can be used through Microsoft.EntityFrameworkCore.EF type’s Functions property:

namespace Microsoft.EntityFrameworkCore

public static class EF

public static DbFunctions Functions { get; }

// Other members.

}

Extension methods are defined for the DbFunctions output type to represent database functions and operators. As fore mentioned, EF Core implements a provider model, so these mapping functions are provides in 2 levels. The EF Core base library provides mapping functions which should be supported by all database providers, like the LIKE operator:

namespace Microsoft.EntityFrameworkCore

public static class DbFunctionsExtensions

public static bool Like(this DbFunctions _, string matchExpression, string pattern);

// Other members.

}

These are also called canonical functions. The mapping funcions for specific database is provided by the database provider library. For example, Microsoft.EntityFrameworkCore.SqlServer.dll library provides DateDiffDay extension method to represent SQL database’s DATEDIFF function for day, and provides Contains extension method to represent SQL database’s CONTAINS function, etc.

namespace Microsoft.EntityFrameworkCore

public static class SqlServerDbFunctionsExtensions

public static bool Contains(this DbFunctions _, string propertyReference, string searchCondition);

public static int DateDiffDay(this DbFunctions _, DateTime startDate, DateTime endDate);

// Other members.

}

The following example filters the product’s names with a pattern. In the following LINQ to Entities query expression tree, the MethodCallExpression node representing Like call is compiled to a LikeExpression node representing the LIKE operator:

internal static void DatabaseOperator(AdventureWorks adventureWorks)

IQueryable<string>products = adventureWorks.Products

.Select(product => product.Name)

.Where(name => EF.Functions.Like(name, "%Touring%50%")); // Define query.

products.WriteLines(); // Execute query.

// SELECT [product].[Name]

// FROM [Production].[Product] AS [product]

// WHERE [product].[Name] LIKE N'%Touring%50%'

}

The following LINQ to Entities query calculates the number of days between current time and photo’s last modified time. In the following LINQ to Entities query expression tree, the MethodCallExpression node representing DateDiffDay call can be compiled to a SqlFunctionExpression node representing DATEDIFF call:

internal static void DatabaseFunction(AdventureWorks adventureWorks)

var photos = adventureWorks.ProductPhotos.Select(photo => new

LargePhotoFileName = photo.LargePhotoFileName,

UnmodifiedDays = EF.Functions.DateDiffDay(photo.ModifiedDate, DateTime.UtcNow)

}); // Define query.

photos.WriteLines(); // Execute query.

// SELECT [photo].[LargePhotoFileName], DATEDIFF(DAY, [photo].[ModifiedDate], GETUTCDATE()) AS [UnmodifiedDays]

// FROM [Production].[ProductPhoto] AS [photo]

}

Database expression tree to database query

With database expression tree, EF can traverse and compile it to SQL query.

SQL generator and SQL command

The SQL database provider of EF Core provides a SQL generator to traverse the compiled database query abstract syntactic tree, and generate SQL database specific remote SQL query. EF Core defines SQL generator as Microsoft.EntityFrameworkCore.Query.Sql.IQuerySqlGenerator interface:

namespace Microsoft.EntityFrameworkCore.Query.Sql

public interface IQuerySqlGenerator

IRelationalCommand GenerateSql(

IReadOnlyDictionary<string, object>parameterValues);

// Other members.

}

It is implemented by Microsoft.EntityFrameworkCore.Query.Sql.Internal.SqlServerQuerySqlGenerator. SQL generator wraps a database expression tree inside, and provides a GenerateSql method, which returns Microsoft.EntityFrameworkCore.Storage.IRelationalCommand to represents generated SQL:

namespace Microsoft.EntityFrameworkCore.Storage

public interface IRelationalCommand

string CommandText { get; }

IReadOnlyList<IRelationalParameter> Parameters { get; }

RelationalDataReader ExecuteReader(

IRelationalConnection connection,

IReadOnlyDictionary<string, object>parameterValues);

// Other members.

}

It is implemented by Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand in Microsoft.EntityFrameworkCore.Relational package.

Generate SQL from database expression tree

The following extension method of DbContext can be defined to accepot a database command tree, and generate SQL:

public static IRelationalCommand Generate(this DbContext dbContext, SelectExpression databaseExpression)

IQuerySqlGeneratorFactory sqlGeneratorFactory = dbContext.GetService<IQuerySqlGeneratorFactory>();

QuerySqlGenerator sqlGenerator = sqlGeneratorFactory.Create();

return sqlGenerator.GetCommand(databaseExpression);

}

The above WhereAndSelectDatabaseExpressions and SelectAndFirstDatabaseExpressions functions build database expression trees from scratch. Take them as an example to generate SQL:

internal static void WhereAndSelectSql(AdventureWorks adventureWorks)

SelectExpression databaseExpression = WhereAndSelectDatabaseExpressions(

adventureWorks);

IRelationalCommand sql = adventureWorks.Generate(

databaseExpression: databaseExpression, parameters: null);

sql.CommandText.WriteLine();

// SELECT [product].[Name]

// FROM [Production].[ProductCategory] AS [product]

// WHERE CAST(LEN([product].[Name]) AS int)> 10

internal static void SelectAndFirstSql(AdventureWorks adventureWorks)

SelectExpression databaseExpression = SelectAndFirstDatabaseExpressions(adventureWorks);

IRelationalCommand sql = adventureWorks.Generate(databaseExpression: databaseExpression, parameters: null);

sql.CommandText.WriteLine();

// SELECT TOP(1) [product].[Name]

// FROM [Production].[Product] AS [product]

}

SQL generator traverses the command tree nodes, a specific Visit overloads is called for each supported node type. It generates SELECT clause from DbProjectionExpression node, FROM clause from DbScanExpression node, WHERE clause from DbFilterExpression node, LIKE operator from DbLikeExpression, etc.

So finally LINQ to Entities queries are translated to remote SQL database queries. The next part discusses the query execution and data loading.

Entity Framework Core and LINQ to Entities in Depth (4) Query Methods (Operators)

Mon, 07 Oct 2019 00:00:00 GMT

[LINQ via C# series]

[Entity Framework Core (EF Core) series]

[Entity Framework (EF) series]

This part discusses how to query SQL database with the defined mapping entities. In EF Core, LINQ to Entities supports most of the standard queries provided by Queryable:

1. Sequence queries: return a new IQueryable<T> source

o Filtering (restriction): Where, OfType*

o Mapping (projection): Select

o Generation: DefaultIfEmpty*

o Grouping: GroupBy*

o Join: Join, GroupJoin, SelectMany, Select

o Concatenation: Concat*

o Set: Distinct, GroupBy*, Union*, Intersect*, Except*

o Convolution: ~Zip~

o Partitioning: Take, Skip, ~TakeWhile~, ~SkipWhile~

o Ordering: OrderBy*, ThenBy, OrderByDescending*, ThenByDescending, ~Reverse~

o Conversion: Cast, AsQueryable

2. Value queries: return a single value

o Element: First, FirstOrDefault, Last*, LastOrDefault*, ~ElementAt~, ~ElementAtOrDefault~, Single, SingleOrDefault

o Aggregation: ~Aggregate~, Count, LongCount, Min, Max, Sum, Average*

o Quantifier: All, Any, Contains

o Equality: ~SequenceEqual~

In above list:

· The crossed queries are not supported by LINQ to Entities (the list provided by MDSN is not up to date), because they cannot be translated to proper SQL database operations. For example, SQL database has no built-in Zip operation support. Calling these crossed queries throws NotSupportedException at runtime

· The underlined queries have some overloads supported by LINQ to Entities, and other overloads not supported:

o For GroupBy, Join, GroupJoin, Distinct, Union, Intersect, Except, Contains, the overloads accepting IEqualityComparer<T> parameter are not supported, because apparently IEqualityComparer<T> has no equivalent SQL translation

o For OrderBy, ThenBy, OrderByDescending, ThenByDescending, the overloads with IComparer<T> parameter are not supported

o For Where, Select, SelectMany, the indexed overloads are not supported

· In EF Core, the queries marked with * can execute the query locally in some cases, without being translated to SQL.

For LINQ to Entities, apparently these queries enable fluent chaining, implement the same LINQ query expression pattern as LINQ to Objects and Parallel LINQ. So in this part, most of the LINQ to Entities queries are demonstrated with queries.

Sequence queries

Similar to the other kinds of LINQ, LINQ to Entities implements deferred execution for these queries returning IQueryable<T>. The SQL query is translated and executed only when trying to pull the result value from IQueryable<T> for the first time.

Filtering (restriction)

EF Core translates Where function call to WHERE clause in SQL, and translates the predicate expression tree (again, not predicate function) to the condition in WHERE clause. The following example queries categories with ProductCategoryID greater than 0:

internal static void Where(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> source = adventureWorks.ProductCategories;

IQueryable<ProductCategory> categories = source.Where(category => category.ProductCategoryID > 0); // Define query.

categories.WriteLines(category => category.Name); // Execute query.

// SELECT [category].[ProductCategoryID], [category].[Name]

// FROM [Production].[ProductCategory] AS [category]

// WHERE [category].[ProductCategoryID] > 0

}

When WriteLines executes, it pulls the results from the query represented by IQueryable<ProductCategory>. At this moment, the query is translated to SQL, and executed in database, then SQL execution results are read by EF Core and yielded.

The C# || operator in the predicate expression tree is translated to SQL OR operator in WHERE clause:

internal static void WhereWithOr(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> source = adventureWorks.ProductCategories;

IQueryable<ProductCategory> categories = source.Where(category =>

category.ProductCategoryID < 2 || category.ProductCategoryID > 3); // Define query.

categories.WriteLines(category => category.Name); // Execute query.

// SELECT [category].[ProductCategoryID], [category].[Name]

// FROM [Production].[ProductCategory] AS [category]

// WHERE ([category].[ProductCategoryID] < 2) OR ([category].[ProductCategoryID] > 3)

}

Similarly, the C# && operator is translated to SQL AND operator:

internal static void WhereWithAnd(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> source = adventureWorks.ProductCategories;

IQueryable<ProductCategory> categories = source.Where(category =>

category.ProductCategoryID > 0 && category.ProductCategoryID < 5); // Define query.

categories.WriteLines(category => category.Name); // Execute query.

// SELECT [category].[ProductCategoryID], [category].[Name]

// FROM [Production].[ProductCategory] AS [category]

// WHERE ([category].[ProductCategoryID] > 0) AND ([category].[ProductCategoryID] < 5)

}

Multiple Where calls are also translated to one single WHERE clause with AND:

internal static void WhereAndWhere(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> source = adventureWorks.ProductCategories;

IQueryable<ProductCategory> categories = source

.Where(category => category.ProductCategoryID > 0)

.Where(category => category.ProductCategoryID < 5); // Define query.

categories.WriteLines(category => category.Name); // Execute query.

// SELECT [category].[ProductCategoryID], [category].[Name]

// FROM [Production].[ProductCategory] AS [category]

// WHERE ([category].[ProductCategoryID] > 0) AND ([category].[ProductCategoryID] < 5)

}

The other filtering query, OfType, can be used for entity types in inheritance hierarchy. And it is equivalent to Where query with is operator. The following examples both query sales transactions from all transactions:

internal static void WhereWithIs(AdventureWorks adventureWorks)

{

IQueryable<TransactionHistory> source = adventureWorks.Transactions;

IQueryable<TransactionHistory> transactions = source.Where(transaction => transaction is SalesTransactionHistory); // Define query.

transactions.WriteLines(transaction => $"{transaction.GetType().Name} {transaction.TransactionDate} {transaction.ActualCost}"); // Execute query.

// SELECT [transaction].[TransactionID], [transaction].[ActualCost], [transaction].[ProductID], [transaction].[Quantity], [transaction].[TransactionDate], [transaction].[TransactionType]

// FROM [Production].[TransactionHistory] AS [transaction]

// WHERE [transaction].[TransactionType] IN (N'W', N'S', N'P') AND ([transaction].[TransactionType] = N'S')

}

internal static void OfTypeEntity(AdventureWorks adventureWorks)

{

IQueryable<TransactionHistory> source = adventureWorks.Transactions;

IQueryable<WorkTransactionHistory> transactions = source.OfType<WorkTransactionHistory>(); // Define query.

transactions.WriteLines(transaction => $"{transaction.GetType().Name} {transaction.TransactionDate} {transaction.ActualCost}"); // Execute query.

// SELECT [t].[TransactionID], [t].[ActualCost], [t].[ProductID], [t].[Quantity], [t].[TransactionDate], [t].[TransactionType]

// FROM [Production].[TransactionHistory] AS [t]

// WHERE [t].[TransactionType] = N'W'

}

When primitive type is specified for OfType, it works locally. The following example queries products with ProductSubcategoryID not null:

internal static void OfTypePrimitive(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

IQueryable<int> products = source.Select(product => product.ProductSubcategoryID).OfType<int>(); // Define query.

products.ToArray().Length.WriteLine(); // Execute query.

// SELECT [p].[ProductSubcategoryID]

// FROM [Production].[Product] AS [p]

}

In EF Core, the above query is translated to a basic SELECT statement without filtering. EF Core executes the translated SQL to query the specified nullable int column of all rows to local, then the int results are locally filtered from all the nullable int results.

Mapping (projection)

In above queries, Queryable.Select is not called, and the query results are entities. So in the translated SQL, the SELECT clause queries all the mapped columns in order to construct the result entities. When Select is called, the selector expression tree is translated into SELECT clause. The following example queries persons’ full names by concatenating the first name and last name:

internal static void Select(AdventureWorks adventureWorks)

{

IQueryable<Person> source = adventureWorks.People;

IQueryable<string> names = source.Select(person =>

person.FirstName + " " + person.LastName); // Define query.

names.WriteLines(); // Execute query.

// SELECT ([person].[FirstName] + N' ') + [person].[LastName]

// FROM [Person].[Person] AS [person]

}

In EF Core, Select also work with anonymous type. For example:

internal static void SelectAnonymousType(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var products = source.Select(product =>

new { Name = product.Name, IsExpensive = product.ListPrice > 1_000 }); // Define query.

products.WriteLines(); // Execute query.

// SELECT [product].[Name], CASE

// WHEN [product].[ListPrice] > 1000.0

// THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)

// END

// FROM [Production].[Product] AS [product]

}

In EF Core, Select supports entity type too:

internal static void SelectEntity(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

IQueryable<Product> products = source

.Where(product => product.ListPrice > 1_000)

.Select(product => new Product()

{

ProductID = product.ProductID,

Name = product.Name

}); // Define query.

products.WriteLines(product => $"{product.ProductID}: {product.Name}"); // Execute query.

// SELECT [product].[ProductID], [product].[Name]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] > 1000.0

}

Generation

As fore mentioned, DefaultIfEmpty is the only built-in generation query:

internal static void DefaultIfEmptyEntity(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> source = adventureWorks.ProductCategories;

IQueryable<ProductCategory> categories = source

.Where(category => category.ProductCategoryID < 0)

.DefaultIfEmpty(); // Define query.

categories.ForEach( // Execute query.

category => (category == null).WriteLine()); // True

// SELECT [t].[ProductCategoryID], [t].[Name]

// FROM (

// SELECT NULL AS [empty]

// ) AS [empty]

// LEFT JOIN (

// SELECT [category].[ProductCategoryID], [category].[Name]

// FROM [Production].[ProductCategory] AS [category]

// WHERE [category].[ProductCategoryID] < 0

// ) AS [t] ON 1 = 1

}

In the above query, Where function call is translated to SQL query with WHERE clause. Since DefaultIfEmpty should yield at least 1 entity, it is translated to LEFT JOIN with a single row table on a condition that always holds, so that the final query result is guaranteed to have at least 1 row. Here Where filters out all entities, in another word, the right table of LEFT JOIN has no rows, so the LEFT JOIN results 1 row, where all columns are NULL, including primary key. Therefore, DefaultIfEmpty yields a null entity. Besides entity type, DefaultIfEmpty works with primitive type in the same way.

The other DefaultIfEmpty overload accepts a specified default value. EF Core does not translate it to SQL, but execute the query logic locally. For example:

internal static void DefaultIfEmptyWithDefaultEntity(AdventureWorks adventureWorks)

{

ProductCategory @default = new ProductCategory() { Name = nameof(ProductCategory) };

IQueryable<ProductCategory> source = adventureWorks.ProductCategories;

IQueryable<ProductCategory> categories = source

.Where(category => category.ProductCategoryID < 0)

.DefaultIfEmpty(@default); ; // Define query.

categories.WriteLines( // Execute query.

category => category?.Name); // ProductCategory

// SELECT [category].[ProductCategoryID], [category].[Name]

// FROM [Production].[ProductCategory] AS [category]

// WHERE [category].[ProductCategoryID] < 0

}

Here the source query for DefaultIfEmpty is translated to SQL and executed, then EF Core reads the results to local, and detect the results locally. If there is no result row, the specified default value is used. DefaultIfEmpty works for specified default primitive value locally too.

internal static void DefaultIfEmptyWithDefaultPrimitive(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> source = adventureWorks.ProductCategories;

IQueryable<int> categories = source

.Where(category => category.ProductCategoryID < 0)

.Select(category => category.ProductCategoryID)

.DefaultIfEmpty(-1); // Define query.

categories.WriteLines(); // Execute query.

// SELECT [category].[ProductCategoryID]

// FROM [Production].[ProductCategory] AS [category]

// WHERE [category].[ProductCategoryID] < 0

}

Notice the default value –1 is translated into the remote SQL query. It is the query result if the right table of left outer join is empty. So there is no local query or local detection executed.

Just like in LINQ to Objects, DefaultIfEmpty can also be used to implement outer join, which is discussed later.

Grouping

When Group query is not used with aggregation query, EF Core executes grouping locally. For example. The following examples group the subcategories by category:

internal static void GroupBy(AdventureWorks adventureWorks)

{

IQueryable<ProductSubcategory> source = adventureWorks.ProductSubcategories;

IQueryable<ProductSubcategory> grouped = source

.GroupBy(keySelector: subcategory => subcategory.ProductCategoryID)

.SelectMany(group => group); // Define query.

grouped.WriteLines(subcategory => subcategory.Name); // Execute query.

// SELECT [subcategory].[ProductSubcategoryID], [subcategory].[Name], [subcategory].[ProductCategoryID]

// FROM [Production].[ProductSubcategory] AS [subcategory]

// ORDER BY [subcategory].[ProductCategoryID]

}

internal static void GroupByWithElementSelector(AdventureWorks adventureWorks)

{

IQueryable<ProductSubcategory> source = adventureWorks.ProductSubcategories;

IQueryable<IGrouping<int, string>> groups = source.GroupBy(

keySelector: subcategory => subcategory.ProductCategoryID,

elementSelector: subcategory => subcategory.Name); // Define query.

groups.WriteLines(group => $"{group.Key}: {string.Join(", ", group)}"); // Execute query.

// SELECT [subcategory].[ProductSubcategoryID], [subcategory].[Name], [subcategory].[ProductCategoryID]

// FROM [Production].[ProductSubcategory] AS [subcategory]

// ORDER BY [subcategory].[ProductCategoryID]

}

EF Core only translates GroupBy an additional ORDER BY clause with the grouping key, so that when reading the SQL execution results to local, the subcategories appears group by group.

When GroupBy is used with supported aggregation query, it is translated to GROUP BY clause. This can be done with a GroupBy overload accepting a result selector, or equivalently an additional Select query. The following examples call aggregation query Count to flatten the results, and they have identical translation:

internal static void GroupByWithResultSelector(AdventureWorks adventureWorks)

{

IQueryable<ProductSubcategory> source = adventureWorks.ProductSubcategories;

var groups = source.GroupBy(

keySelector: subcategory => subcategory.ProductCategoryID,

elementSelector: subcategory => subcategory.Name,

resultSelector: (key, group) => new { CategoryID = key, SubcategoryCount = group.Count() }); // Define query.

groups.WriteLines(); // Execute query.

// SELECT [subcategory].[ProductCategoryID] AS [CategoryID], COUNT(*) AS [SubcategoryCount]

// FROM [Production].[ProductSubcategory] AS [subcategory]

// GROUP BY [subcategory].[ProductCategoryID]

}

internal static void GroupByAndSelect(AdventureWorks adventureWorks)

{

IQueryable<ProductSubcategory> source = adventureWorks.ProductSubcategories;

var groups = source

.GroupBy(

keySelector: subcategory => subcategory.ProductCategoryID,

elementSelector: subcategory => subcategory.Name)

.Select(group => new { CategoryID = group.Key, SubcategoryCount = group.Count() }); // Define query.

groups.WriteLines(); // Execute query.

// SELECT [subcategory].[ProductCategoryID] AS [CategoryID], COUNT(*) AS [SubcategoryCount]

// FROM [Production].[ProductSubcategory] AS [subcategory]

// GROUP BY [subcategory].[ProductCategoryID]

}

GroupBy’s key selector can return anonymous type with multiple properties to support grouping by multiple keys:

internal static void GroupByMultipleKeys(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var groups = source

.GroupBy(

keySelector: product => new

{

ProductSubcategoryID = product.ProductSubcategoryID,

ListPrice = product.ListPrice

resultSelector: (key, group) => new

{

ProductSubcategoryID = key.ProductSubcategoryID,

ListPrice = key.ListPrice,

Count = group.Count()

})

.Where(group => group.Count > 1); // Define query.

groups.WriteLines(); // Execute query.

// SELECT [product].[ProductSubcategoryID], [product].[ListPrice], COUNT(*) AS [Count]

// FROM [Production].[Product] AS [product]

// GROUP BY [product].[ProductSubcategoryID], [product].[ListPrice]

// HAVING COUNT(*) > 1

}

The additional Where query is translated to HAVING clause, as expected.

Join

Inner join

Similar to LINQ to Objects, Join is provided for inner join. The following example simply join the subcategories and categories with foreign key:

internal static void InnerJoinWithJoin(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

IQueryable<ProductSubcategory> inner = adventureWorks.ProductSubcategories;

var categorySubcategories = outer.Join(

inner: inner,

outerKeySelector: category => category.ProductCategoryID,

innerKeySelector: subcategory => subcategory.ProductCategoryID,

resultSelector: (category, subcategory) =>

new { Category = category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// join subcategory in inner

// on category.ProductCategoryID equals subcategory.ProductCategoryID

// select new { Category = category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

// SELECT [category].[Name], [subcategory].[Name]

// FROM [Production].[ProductCategory] AS [category]

// INNER JOIN [Production].[ProductSubcategory] AS [subcategory] ON [category].[ProductCategoryID] = [subcategory].[ProductCategoryID]

}

Join’s key selectors can return anonymous type to join with multiple keys:

internal static void InnerJoinWithMultipleKeys(AdventureWorks adventureWorks)

{

IQueryable<Product> outer = adventureWorks.Products;

IQueryable<TransactionHistory> inner = adventureWorks.Transactions;

var transactions = outer.Join(

inner: inner,

outerKeySelector: product =>

new { ProductID = product.ProductID, UnitPrice = product.ListPrice },

innerKeySelector: transaction =>

new { ProductID = transaction.ProductID, UnitPrice = transaction.ActualCost / transaction.Quantity },

resultSelector: (product, transaction) =>

new { Name = product.Name, Quantity = transaction.Quantity }); // Define query.

// var transactions =

// from product in adventureWorks.Products

// join transaction in adventureWorks.Transactions

// on new { ProductID = product.ProductID, UnitPrice = product.ListPrice }

// equals new { ProductID = transaction.ProductID, UnitPrice = transaction.ActualCost / transaction.Quantity }

// select new { Name = product.Name, Quantity = transaction.Quantity };

transactions.WriteLines(); // Execute query.

// SELECT [product].[Name], [transaction].[Quantity]

// FROM [Production].[Product] AS [product]

// INNER JOIN [Production].[TransactionHistory] AS [transaction] ON ([product].[ProductID] = [transaction].[ProductID]) AND ([product].[ListPrice] = ([transaction].[ActualCost] / [transaction].[Quantity]))

// WHERE [transaction].[TransactionType] IN (N'W', N'S', N'P')

}

Just like LINQ to Objects, inner join can be done by SelectMany, Select, and GroupJoin as well. In the following example, Select returns hierarchical data, so an additional SelectMany can flatten the result:

internal static void InnerJoinWithSelect(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

IQueryable<ProductSubcategory> inner = adventureWorks.ProductSubcategories;

var categorySubcategories = outer

.Select(category => new

{

Category = category,

Subcategories = inner

.Where(subcategory => category.ProductCategoryID == subcategory.ProductCategoryID)

// LEFT OUTER JOIN if DefaultIfEmpty is called.

})

.SelectMany(

collectionSelector: category => category.Subcategories,

resultSelector: (category, subcategory) =>

new { Category = category.Category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// select new

// {

// Category = category,

// Subcategories = from subcategory in inner

// where category.ProductCategoryID == subcategory.ProductCategoryID

// select subcategory

// } into category

// from subcategory in category.Subcategories

// select new { Category = category.Category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

// SELECT [category].[Name], [subcategory].[Name]

// FROM [Production].[ProductCategory] AS [category]

// CROSS JOIN [Production].[ProductSubcategory] AS [subcategory]

// WHERE [category].[ProductCategoryID] = [subcategory].[ProductCategoryID]

}

EF Core translates the above query to CROOS JOIN with WHERE clause, which is equivalent to the previous INNER JOIN query, with the same query plan.

The following example implement the same inner join directly with SelectMany. Its SQL translation is the same INNER JOIN as the first Join example:

internal static void InnerJoinWithSelectMany(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

IQueryable<ProductSubcategory> inner = adventureWorks.ProductSubcategories;

var categorySubcategories = outer

.SelectMany(

collectionSelector: category => inner

.Where(subcategory => category.ProductCategoryID == subcategory.ProductCategoryID),

// LEFT OUTER JOIN if DefaultIfEmpty is called.

resultSelector: (category, subcategory) =>

new { Category = category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// from subcategory in (from subcategory in inner

// where category.ProductCategoryID == subcategory.ProductCategoryID

// select subcategory)

// select new { Category = category.Name, Subcategory = subcategory.Name };

// Or equivalently:

// var categorySubcategories =

// from category in outer

// from subcategory in inner

// where category.ProductCategoryID == subcategory.ProductCategoryID

// select new { Category = category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

}

The above Select and SelectMany has a Where subquery to filter the related entities to join with. The Where subquery can be substituted by collection navigation property. After the substitution, the queries are translated to the same INNER JOIN as the first Join example:

internal static void InnerJoinWithSelectAndRelationship(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

var categorySubcategories = outer

.Select(category => new { Category = category, Subcategories = category.ProductSubcategories })

.SelectMany(

collectionSelector: category => category.Subcategories,

// LEFT OUTER JOIN if DefaultIfEmpty is missing.

resultSelector: (category, subcategory) =>

new { Category = category.Category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// select new { Category = category, Subcategories = category.ProductSubcategories } into category

// from subcategory in category.Subcategories

// select new { Category = category.Category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

}

internal static void InnerJoinWithSelectManyAndRelationship(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

var categorySubcategories = outer.SelectMany(

collectionSelector: category => category.ProductSubcategories,

// LEFT OUTER JOIN if DefaultIfEmpty is missing.

resultSelector: (category, subcategory) =>

new { Category = category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// from subcategory in category.ProductSubcategories

// select new { Category = category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

}

GroupJoin also returns hierarchical result, so again an additional SelectMany can flatten the result. The following example still has the same INNER JOIN translation as the first Join example:

internal static void InnerJoinWithGroupJoinAndSelectMany(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

IQueryable<ProductSubcategory> inner = adventureWorks.ProductSubcategories;

var categorySubcategories = outer

.GroupJoin(

inner: inner,

outerKeySelector: category => category.ProductCategoryID,

innerKeySelector: subcategory => subcategory.ProductCategoryID,

resultSelector: (category, subcategories) =>

new { Category = category, Subcategories = subcategories })

.SelectMany(

collectionSelector: category => category.Subcategories,

// LEFT OUTER JOIN if DefaultIfEmpty is called.

resultSelector: (category, subcategory) =>

new { Category = category.Category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// join subcategory in inner

// on category.ProductCategoryID equals subcategory.ProductCategoryID into subcategories

// from subcategory in subcategories

// select new { Category = category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

}

Navigation property makes it very easy to join entities with relationship. The following example inner joins 3 entity types, where 2 entity types have many-to-many relationship with a junction entity type:

internal static void MultipleInnerJoinsWithRelationship(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var productPhotos = source.SelectMany(

collectionSelector: product => product.ProductProductPhotos,

resultSelector: (product, productProductPhoto) => new

{

Product = product.Name,

Photo = productProductPhoto.ProductPhoto.LargePhotoFileName

}); // Define query.

// var productPhotos =

// from product in source

// from productProductPhoto in product.ProductProductPhotos

// select new { Product = product.Name, Photo = productProductPhoto.ProductPhoto.LargePhotoFileName };

productPhotos.WriteLines(); // Execute query.

// SELECT [product].[Name], [product.ProductProductPhotos.ProductPhoto].[LargePhotoFileName]

// FROM [Production].[Product] AS [product]

// INNER JOIN [Production].[ProductProductPhoto] AS [product.ProductProductPhotos] ON [product].[ProductID] = [product.ProductProductPhotos].[ProductID]

// INNER JOIN [Production].[ProductPhoto] AS [product.ProductProductPhotos.ProductPhoto] ON [product.ProductProductPhotos].[ProductPhotoID] = [product.ProductProductPhotos.ProductPhoto].[ProductPhotoID]

}

Left outer join

GroupJoin is provided for left outer join. The following example have categories to left outer join subcategories with foreign key, and the results have all categories with or without matching subcategories. It is translated to LEFT JOIN:

internal static void LeftOuterJoinWithGroupJoin(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

IQueryable<ProductSubcategory> inner = adventureWorks.ProductSubcategories;

var categorySubcategories = outer

.GroupJoin(

inner: inner,

outerKeySelector: category => category.ProductCategoryID,

innerKeySelector: subcategory => subcategory.ProductCategoryID,

resultSelector: (category, subcategories) =>

new { Category = category, Subcategories = subcategories }); // Define query.

// var categorySubcategories =

// from category in outer

// join subcategory in inner

// on category.ProductCategoryID equals subcategory.ProductCategoryID into subcategories

// select new { Category = category, Subcategories = subcategories };

categorySubcategories.WriteLines(categorySubcategory =>

$@"{categorySubcategory.Category.Name}: {string.Join(

", ", categorySubcategory.Subcategories.Select(subcategory => subcategory.Name))}"); // Execute query.

// SELECT [category].[ProductCategoryID], [category].[Name], [subcategory].[ProductSubcategoryID], [subcategory].[Name], [subcategory].[ProductCategoryID]

// FROM [Production].[ProductCategory] AS [category]

// LEFT JOIN [Production].[ProductSubcategory] AS [subcategory] ON [category].[ProductCategoryID] = [subcategory].[ProductCategoryID]

// ORDER BY [category].[ProductCategoryID]

}

GroupJoin returns hierarchical results. So here the translated SQL also sorts the result by the key, so that EF Core can read the query results group by group. To have flattened results from GroupJoin, SelectMany can be called. As discussed in the LINQ to Objects chapter, an DefaultIfEmpty subquery is required (It becomes inner join if DefaultIfEmpty is missing). The following example has the same SQL translation as above, it just yields result by result instead of group by group.

internal static void LeftOuterJoinWithGroupJoinAndSelectMany(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

IQueryable<ProductSubcategory> inner = adventureWorks.ProductSubcategories;

var categorySubcategories = outer

.GroupJoin(

inner: inner,

outerKeySelector: category => category.ProductCategoryID,

innerKeySelector: subcategory => subcategory.ProductCategoryID,

resultSelector: (category, subcategories) =>

new { Category = category, Subcategories = subcategories }) // Define query.

.SelectMany(

collectionSelector: category => category.Subcategories

.DefaultIfEmpty(), // INNER JOIN if DefaultIfEmpty is missing.

resultSelector: (category, subcategory) =>

new { Category = category.Category, Subcategory = subcategory }); // Define query.

// var categorySubcategories =

// from category in outer

// join subcategory in inner

// on category.ProductCategoryID equals subcategory.ProductCategoryID into subcategories

// from subcategory in subcategories.DefaultIfEmpty()

// select new { Category = category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(categorySubcategory =>

$"{categorySubcategory.Category.Name} {categorySubcategory.Subcategory?.Name}"); // Execute query.

}

Similar to inner join, left outer join can be done with Select and SelectMany too, with a DefaultIfEmpty subquery. The following queries have the same SQL translation:

internal static void LeftOuterJoinWithSelect(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

IQueryable<ProductSubcategory> inner = adventureWorks.ProductSubcategories;

var categorySubcategories = outer

.Select(category => new

{

Category = category,

Subcategories = inner

.Where(subcategory => category.ProductCategoryID == subcategory.ProductCategoryID)

})

.SelectMany(

collectionSelector: category => category.Subcategories

.DefaultIfEmpty(), // INNER JOIN if DefaultIfEmpty is missing.

resultSelector: (category, subcategory) =>

new { Category = category.Category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// select new

// {

// Category = category,

// Subcategories = from subcategory in inner

// where subcategory.ProductCategoryID == category.ProductCategoryID

// select subcategory

// } into category

// from subcategory in category.Subcategories.DefaultIfEmpty()

// select new { Category = category.Category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

// SELECT [category].[Name], [t1].[Name]

// FROM [Production].[ProductCategory] AS [category]

// CROSS APPLY (

// SELECT [t0].*

// FROM (

// SELECT NULL AS [empty]

// ) AS [empty0]

// LEFT JOIN (

// SELECT [subcategory0].*

// FROM [Production].[ProductSubcategory] AS [subcategory0]

// WHERE [category].[ProductCategoryID] = [subcategory0].[ProductCategoryID]

// ) AS [t0] ON 1 = 1

// ) AS [t1]

}

internal static void LeftOuterJoinWithSelectMany(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

IQueryable<ProductSubcategory> inner = adventureWorks.ProductSubcategories;

var categorySubcategories = outer

.SelectMany(

collectionSelector: category => inner

.Where(subcategory => category.ProductCategoryID == subcategory.ProductCategoryID)

.DefaultIfEmpty(), // INNER JOIN if DefaultIfEmpty is missing.

resultSelector: (category, subcategory) =>

new { Category = category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// from subcategory in (from subcategory in inner

// where category.ProductCategoryID == subcategory.ProductCategoryID

// select subcategory).DefaultIfEmpty()

// select new { Category = category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

}

In EF Core, the above 2 queries are both translated to CROSS APPLY, but this is logically equivalent to LEFT JOIN of the GroupJoin example.

As demonstrated for inner join, in the above Select and SelectMany queries, the Where subquery is equivalent to collection navigation property. EF Core support collection navigation property for left outer join with Select and SelectMany. The following queries are translated to the same LEFT JOIN query:

internal static void LeftOuterJoinWithSelectAndRelationship(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

var categorySubcategories = outer

.Select(category => new { Category = category, Subcategories = category.ProductSubcategories })

.SelectMany(

collectionSelector: category => category.Subcategories

.DefaultIfEmpty(), // INNER JOIN if DefaultIfEmpty is missing.

resultSelector: (category, subcategory) =>

new { Category = category.Category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// select new { Category = category, Subcategories = category.ProductSubcategories } into category

// from subcategory in category.Subcategories.DefaultIfEmpty()

// select new { Category = category.Category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

// SELECT [category].[Name] AS [Category], [category.ProductSubcategories].[Name] AS [Subcategory]

// FROM [Production].[ProductCategory] AS [category]

// LEFT JOIN [Production].[ProductSubcategory] AS [category.ProductSubcategories] ON [category].[ProductCategoryID] = [category.ProductSubcategories].[ProductCategoryID]

}

internal static void LeftOuterJoinWithSelectManyAndRelationship(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> outer = adventureWorks.ProductCategories;

var categorySubcategories = outer.SelectMany(

collectionSelector: category => category.ProductSubcategories

.DefaultIfEmpty(), // INNER JOIN if DefaultIfEmpty is missing.

resultSelector: (category, subcategory) =>

new { Category = category.Name, Subcategory = subcategory.Name }); // Define query.

// var categorySubcategories =

// from category in outer

// from subcategory in category.ProductSubcategories.DefaultIfEmpty()

// select new { Category = category.Name, Subcategory = subcategory.Name };

categorySubcategories.WriteLines(); // Execute query.

}

Cross join

Just like LINQ to Objects, cross join can be done with SelectMany and Join. The following example queries the expensive products (list price greater than 2000) and cheap products (list price less than 100), and then cross join them to get all possible product bundles, where each bundle has one expensive product and one cheap product:

internal static void CrossJoinWithSelectMany(AdventureWorks adventureWorks)

{

IQueryable<Product> outer = adventureWorks.Products.Where(product => product.ListPrice > 2000);

IQueryable<Product> inner = adventureWorks.Products.Where(product => product.ListPrice < 100);

var bundles = outer.SelectMany(

collectionSelector: expensiveProduct => inner,

resultSelector: (expensiveProduct, cheapProduct) =>

new { Expensive = expensiveProduct.Name, Cheap = cheapProduct.Name }); // Define query.

// var bundles =

// from outerProduct in outer

// from innerProduct in inner

// select new { Expensive = outerProduct.Name, Cheap = innerProduct.Name };

bundles.WriteLines(); // Execute query.

// SELECT [product].[Name], [product0].[Name]

// FROM [Production].[Product] AS [product]

// CROSS JOIN [Production].[Product] AS [product0]

// WHERE ([product].[ListPrice] > 2000.0) AND ([product0].[ListPrice] < 100.0)

}

The following implementation with Join is equivalent, just have the 2 key selectors always return equal values:

internal static void CrossJoinWithJoin(AdventureWorks adventureWorks)

{

IQueryable<Product> outer = adventureWorks.Products.Where(product => product.ListPrice > 2000);

IQueryable<Product> inner = adventureWorks.Products.Where(product => product.ListPrice < 100);

var bundles = outer.Join(

inner: inner,

outerKeySelector: product => 1,

innerKeySelector: product => 1,

resultSelector: (outerProduct, innerProduct) =>

new { Expensive = outerProduct.Name, Cheap = innerProduct.Name }); // Define query.

// var bundles =

// from outerProduct in outer

// join innerProduct in inner

// on 1 equals 1

// select new { Expensive = outerProduct.Name, Cheap = innerProduct.Name };

bundles.WriteLines(); // Execute query.

// SELECT [product].[Name], [t].[Name]

// FROM [Production].[Product] AS [product]

// INNER JOIN (

// SELECT [product1].*

// FROM [Production].[Product] AS [product1]

// WHERE [product1].[ListPrice] < 100.0

// ) AS [t] ON 1 = 1

// WHERE [product].[ListPrice] > 2000.0

}

It is translated to INNER JOIN, which is equivalent to previous CROSS JOIN, with the same query plan.

Concatenation

The following example concatenates the cheap products and the expensive products, and query the products’ names:

internal static void ConcatEntity(AdventureWorks adventureWorks)

{

IQueryable<Product> first = adventureWorks.Products.Where(product => product.ListPrice < 100);

IQueryable<Product> second = adventureWorks.Products.Where(product => product.ListPrice > 2000);

IQueryable<string> concat = first

.Concat(second)

.Select(product => product.Name); // Define query.

concat.WriteLines(); // Execute query.

// SELECT [product1].[ProductID], [product1].[ListPrice], [product1].[Name], [product1].[ProductSubcategoryID], [product1].[RowVersion]

// FROM [Production].[Product] AS [product1]

// WHERE [product1].[ListPrice] < 100.0

// SELECT [product2].[ProductID], [product2].[ListPrice], [product2].[Name], [product2].[ProductSubcategoryID], [product2].[RowVersion]

// FROM [Production].[Product] AS [product2]

// WHERE [product2].[ListPrice] > 2000.0

}

EF Core supports Concat for primitive type locally as well. In the above example, Select is called after Concat. It is logically equivalent to call Select before Concat, which works in EF Core:

internal static void ConcatPrimitive(AdventureWorks adventureWorks)

{

IQueryable<string> first = adventureWorks.Products

.Where(product => product.ListPrice < 100)

.Select(product => product.Name);

IQueryable<string> second = adventureWorks.Products

.Where(product => product.ListPrice > 2000)

.Select(product => product.Name);

IQueryable<string> concat = first.Concat(second); // Define query.

concat.WriteLines(); // Execute query.

// SELECT [product].[Name]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] < 100.0

// SELECT [product0].[Name]

// FROM [Production].[Product] AS [product0]

// WHERE [product0].[ListPrice] > 2000.0

}

EF Core translates Concat’s 2 data sources to 2 SQL queries, reads the query results to local, and concatenates them locally.

Set

Distinct works with entity type and primitive type. It is translated to the DISTINCT keyword:

internal static void DistinctEntity(AdventureWorks adventureWorks)

{

IQueryable<ProductSubcategory> source = adventureWorks.ProductSubcategories;

IQueryable<ProductCategory> distinct = source

.Select(subcategory => subcategory.ProductCategory)

.Distinct(); // Define query.

distinct.WriteLines(category => $"{category.ProductCategoryID}: {category.Name}"); // Execute query.

// SELECT DISTINCT [subcategory.ProductCategory].[ProductCategoryID], [subcategory.ProductCategory].[Name]

// FROM [Production].[ProductSubcategory] AS [subcategory]

// INNER JOIN [Production].[ProductCategory] AS [subcategory.ProductCategory] ON [subcategory].[ProductCategoryID] = [subcategory.ProductCategory].[ProductCategoryID]

}

internal static void DistinctPrimitive(AdventureWorks adventureWorks)

{ IQueryable<ProductSubcategory> source = adventureWorks.ProductSubcategories;

IQueryable<int> distinct = source

.Select(subcategory => subcategory.ProductCategoryID)

.Distinct(); // Define query.

distinct.WriteLines(); // Execute query.

// SELECT DISTINCT [subcategory].[ProductCategoryID]

// FROM [Production].[ProductSubcategory] AS [subcategory]

}

GroupBy returns groups with distinct keys, so in theory it can be used to query the same result as Distinct:

internal static void DistinctWithGroupBy(AdventureWorks adventureWorks)

{

IQueryable<ProductSubcategory> source = adventureWorks.ProductSubcategories;

IQueryable<int> distinct = source.GroupBy(

keySelector: subcategory => subcategory.ProductCategoryID,

resultSelector: (key, group) => key); // Define query.

distinct.WriteLines(); // Execute query.

// SELECT [subcategory].[ProductCategoryID] AS [Key]

// FROM [Production].[ProductSubcategory] AS [subcategory]

// GROUP BY [subcategory].[ProductCategoryID]

}

However, as fore mentioned, in EF Core, GroupBy executes locally. The above example only queries grouping keys, however it reads all rows of the table to local, which can be a performance issue.

GroupBy can also be used for more complex scenarios. The following example queries the full product entities with distinct list price:

internal static void DistinctWithGroupByAndFirstOrDefault(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

IQueryable<Product> distinct = source.GroupBy(

keySelector: product => product.ListPrice,

resultSelector: (key, group) => group.FirstOrDefault()); // Define query.

distinct.WriteLines(); // Execute query.

// SELECT [product].[ProductID], [product].[ListPrice], [product].[Name], [product].[ProductSubcategoryID]

// FROM [Production].[Product] AS [product]

// ORDER BY [product].[ListPrice]

}

Again, EF Core does not translate grouping to SQL. In this example, only 1 entities for each key is queried, but EF Core reads all rows to local, and execute the grouping logic locally.

EF Core supports Union for entity and primitive types locally.

internal static void UnionEntity(AdventureWorks adventureWorks)

{

IQueryable<Product> first = adventureWorks.Products

.Where(product => product.ListPrice > 100);

IQueryable<Product> second = adventureWorks.Products

.Where(product => product.ProductSubcategoryID == 1);

IQueryable<Product> union = first.Union(second); // Define query.

union.WriteLines(); // Execute query.

// SELECT [product].[ProductID], [product].[ListPrice], [product].[Name], [product].[ProductSubcategoryID]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] > 100.0

// SELECT [product].[ProductID], [product].[ListPrice], [product].[Name], [product].[ProductSubcategoryID]

// FROM [Production].[Product] AS [product]

// [product0].[ProductSubcategoryID] = 1

}

internal static void UnionPrimitive(AdventureWorks adventureWorks)

{

var first = adventureWorks.Products

.Where(product => product.ListPrice > 100)

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice });

var second = adventureWorks.Products

.Where(product => product.ProductSubcategoryID == 1)

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice });

var union = first.Union(second); // Define query.

union.WriteLines(); // Execute query.

// SELECT [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] > 100.0

// SELECT [product0].[Name], [product0].[ListPrice]

// FROM [Production].[Product] AS [product0]

// WHERE [product0].[ProductSubcategoryID] = 1

}

EF Core executes Intersect and Except locally as well.

internal static void IntersectEntity(AdventureWorks adventureWorks)

{

IQueryable<Product> first = adventureWorks.Products

.Where(product => product.ListPrice > 100);

IQueryable<Product> second = adventureWorks.Products

.Where(product => product.ListPrice < 2000);

IQueryable<Product> intersect = first.Intersect(second); // Define query.

intersect.WriteLines(); // Execute query.

// SELECT [product0].[ProductID], [product0].[ListPrice], [product0].[Name], [product0].[ProductSubcategoryID]

// FROM [Production].[Product] AS [product0]

// WHERE [product0].[ListPrice] < 2000.0

// SELECT [product].[ProductID], [product].[ListPrice], [product].[Name], [product].[ProductSubcategoryID]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] > 100.0

}

internal static void ExceptPrimitive(AdventureWorks adventureWorks)

{

var first = adventureWorks.Products

.Where(product => product.ListPrice > 100)

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice });

var second = adventureWorks.Products

.Where(product => product.ListPrice > 2000)

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice });

var except = first.Except(second); // Define query.

except.WriteLines(); // Execute query.

// SELECT [product0].[Name], [product0].[ListPrice]

// FROM [Production].[Product] AS [product0]

// WHERE [product0].[ListPrice] > 2000.0

// SELECT [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] > 100.0

}

Partitioning

Skip is translate to OFFSET filter:

internal static void Skip(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

IQueryable<string> names = source

.Select(product => product.Name)

.Skip(10); // Define query.

names.WriteLines(); // Execute query.

// exec sp_executesql N'SELECT [product].[Name]

// FROM [Production].[Product] AS [product]

// ORDER BY (SELECT 1)

// OFFSET @__p_0 ROWS',N'@__p_0 int',@__p_0=10

}

In SQL, OFFSET is considered to be a part of the ORDER BY clause, so here EF Core generates ORDERBY (SELECT 1) clause.

When Take is called without Skip, it is translate to TOP filter:

internal static void Take(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

IQueryable<string> products = source

.Take(10)

.Select(product => product.Name); // Define query.

products.WriteLines(); // Execute query.

// exec sp_executesql N'SELECT TOP(@__p_0) [product].[Name]

// FROM [Production].[Product] AS [product]',N'@__p_0 int',@__p_0=10

}

When Take is called with Skip, they are translated to FETCH and OFFSET filters:

internal static void SkipAndTake(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

IQueryable<string> products = source

.OrderBy(product => product.Name)

.Skip(20)

.Take(10)

.Select(product => product.Name); // Define query.

products.WriteLines(); // Execute query.

// exec sp_executesql N'SELECT [product].[Name]

// FROM [Production].[Product] AS [product]

// ORDER BY (SELECT 1)

// OFFSET @__p_0 ROWS FETCH NEXT @__p_1 ROWS ONLY',N'@__p_0 int,@__p_1 int',@__p_0=20,@__p_1=10

}

Ordering

OrderBy/OrderByDescending are translated to ORDER BY clause with without/with DESC, for example:

internal static void OrderBy(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var products = source

.OrderBy(product => product.ListPrice)

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice }); // Define query.

products.WriteLines(); // Execute query.

// SELECT [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// ORDER BY [product].[ListPrice]

}

internal static void OrderByDescending(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var products = source

.OrderByDescending(product => product.ListPrice)

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice }); // Define query.

products.WriteLines(); // Execute query.

// SELECT [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// ORDER BY [product].[ListPrice] DESC

}

To sort with multiple keys, call OrderBy/OrderByDescending and ThenBy/ThenByDescending:

internal static void OrderByAndThenBy(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var products = source

.OrderBy(product => product.ListPrice)

.ThenBy(product => product.Name)

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice }); // Define query.

products.WriteLines(); // Execute query.

// SELECT [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// ORDER BY [product].[ListPrice], [product].[Name]

}

In EF Core, when the key selector returns anonymous type to sort by multiple keys, the sorting is executed locally:

internal static void OrderByMultipleKeys(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var products = source

.OrderBy(product => new { ListPrice = product.ListPrice, Name = product.Name })

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice }); // Define query.

products.WriteLines(); // Execute query.

// SELECT [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// ORDER BY (SELECT 1)

}

Multiple OrderBy/OrderByDescending calls are translated to SQL reversely. The following example sort all products by list price, then sort all products again by subcategory, which is equivalent to sort all products by subcategory first, then sort products in the same subcategory by list price:

internal static void OrderByAndOrderBy(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var products = source

.OrderBy(product => product.ListPrice)

.OrderBy(product => product.ProductSubcategoryID)

.Select(product => new

{

Name = product.Name,

ListPrice = product.ListPrice,

Subcategory = product.ProductSubcategoryID

}); // Define query.

products.WriteLines(); // Execute query.

// SELECT [product].[Name], [product].[ListPrice], [product].[ProductSubcategoryID]

// FROM [Production].[Product] AS [product]

// ORDER BY [product].[ProductSubcategoryID], [product].[ListPrice]

}

Conversion

Cast can work with entity type. The following example casts base entity to derived entity:

internal static void CastEntity(AdventureWorks adventureWorks)

{

IQueryable<TransactionHistory> source = adventureWorks.Transactions;

IQueryable<TransactionHistory> transactions = source

.Where(product => product.ActualCost > 500)

.Cast<SalesTransactionHistory>(); // Define query.

transactions.WriteLines(transaction =>

$"{transaction.GetType().Name}: {transaction.TransactionDate}"); // Execute query.

// SELECT [product].[TransactionID], [product].[ActualCost], [product].[ProductID], [product].[Quantity], [product].[TransactionDate], [product].[TransactionType]

// FROM [Production].[TransactionHistory] AS [product]

// WHERE [product].[TransactionType] IN (N'W', N'S', N'P') AND ([product].[ActualCost] > 500.0)

}

EF Core does not support Cast for primitive type.

Queryable has an additional query, AsQueryable, which accepts IEnumerable<T> and returns IQueryable<T>. Remember Enumerable.AsEnumerable can convert more derived sequence (like List<T>, IQueryable<T>, etc.) to IEnumerable<T>. So the Queryable.AsQueryable/Eumerable.AsEnumerable queries look similar to the ParallelEnumerable.AsParallel/ParallelEnumerable.AsSequential queries, which convert between sequential and parallel local queries at any point. However, AsQueryable/AsEnumerable usually do not convert freely between local and remote queries. The following is the implementation of AsEnumerable and AsQueryable:

namespace System.Linq

{

public static class Enumerable

{

public static IEnumerable<TSource> AsEnumerable<TSource>(this IEnumerable<TSource> source) => source;

}

public static class Queryable

{

public static IQueryable<TElement> AsQueryable<TElement>(this IEnumerable<TElement> source) =>

source as IQueryable<TElement> ?? new EnumerableQuery<TElement>(source);

}

AsQueryable accepts an IEnumerable<T> source. If the source is indeed an IQueryable<T> source, then do nothing and just return it; if not, wrap the source into an System.Linq.EnumerableQuery<T> instance, and return it. EnumerableQuery<T> is a special implementation of IQueryable<T>. If an IQueryable<T> query is an EnumerableQuery<T> instance, when this query is executed, it internally calls System.Linq.EnumerableRewriter to translate itself to local query, then execute the translated query locally. For example, AdventureWorks.Products return IQueryable<Product>, which is actually a DbSet<T> instance, so calling AsQueryable with AdventureWorks.Products does nothing and returns the DbSet<Product> instance itself, which can have its subsequent queries to be translated to SQL by EF Core. In contrast, calling AsQueryable with a T[] array returns an EnumerableQuery<T> wrapper, which is a local mocking of remote query and can have its subsequent queries to be translated to local queries, As a result, AsEnumerable can always convert a remote LINQ to Entities query to local LINQ to Objects query, but AsQueryable cannot always convert arbitrary local LINQ to Objects query to a remote LINQ to Entities query (and logically, an arbitrary local .NET data source cannot be converted to a remote data source like SQL database). For example:

internal static void AsEnumerableAsQueryable(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var remoteAndLocal = source // DbSet<T>.

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice }) // Return EntityQueryable<T>.

.AsEnumerable() // Do nothing. Directly return the EntityQueryable<T> source.

.Where(product => product.ListPrice > 0) // Enumerable.Where. Return a generator wrapping the EntityQueryable<T> source.

.AsQueryable() // Return an EnumerableQuery<T> instance wrapping the source generator.

.OrderBy(product => product.Name); // Queryable.OrderBy. Return EnumerableQuery<T>.

remoteAndLocal.WriteLines();

// SELECT [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

var remote = source // DbSet<T>.

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice }) // Return EntityQueryable<T>.

.AsEnumerable() // Do nothing. Directly return the EntityQueryable<T> source.

.AsQueryable() // Do nothing. Directly return the EntityQueryable<T> source.

.Where(product => product.ListPrice > 0) // Still LINQ to Entities. Return EntityQueryable<T>.

.OrderBy(product => product.Name); // Still LINQ to Entities. Return EntityQueryable<T>.

remote.WriteLines();

// SELECT [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] > 0.0

// ORDER BY [product].[Name]

}

In the first query, the LINQ to Entities source is chained with Select, then AsEnumerable returns IEnumerable<T>, so the following Where is Enumerable.Where, and it returns a generator. Then AsQueryable detects if the generator is IQueryable<T>. Since the generator is not IQueryable<T>, AsQueryable returns a EnumerableQuery<T> wrapper, which can have the following OrderBy translated to local query. So in this entire query chaining, only Select, which is before AsEnumerable, can be translated to SQL and executed remotely, all the other queries are executed locally.

· The source is a DbSet<T> instance, which implements IQueryable<T> and represents the LINQ to Entities data source - rows in remote SQL database table.

· Queryable.Select is called on DbSet<T> source, in this case it returns a Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable<T> instance in EF Core, which implements IQueryable<T> and represents LINQ to Entities query.

· Enumerable.AsEnumerable does nothing and directly returns its source, the EntityQueryable<T> instance

· Enumerable.Where is called, since AsEnumerable returns IEnumerable<T> type. Where returns a generator wrapping its source, the EntityQueryable<T> instance.

· Queryable.AsQueryable is called. Its source, the generator from Where, implements IEnumerable<T>, not IQueryable<T>, so AsQueryable return an EnumerableQuery<T> instance wrapping the generator. As fore mentioned, EnumerableQuery<T> has nothing to do with database.

· Queryable.OrderBy is called with EnumerableQuery<T> instance, in this case it returns another EnumerableQuery<T> instance, which has nothing to do with database either.

So the first query is a hybrid query. When it is executed, only Select is remote LINQ to Entities query and is translated to SQL. After AsEnumerable, Where goes local, then AsQueryable cannot convert back to remote LINQ to Entities query anymore. So, Where and OrderBy are both local queries, and not translated to SQL.

The second query is a special case, where AsEnumerable is chained with AsQueryable right away. In this case, AsEnumerable and AsQueryable both do nothing at all. The following Where and OrderBy are both LINQ to Entities queries, and translated to SQL along with Select.

Value query

Queries in this category accepts an IQueryable<T> source and returns a single value. As fore mentioned, the aggregation queries can be used with GroupBy. When value queries are called at the end of a LINQ to Entities query, they executes the query immediately.

Element

First and FirstOrDefault execute the LINQ to Entities queries immediately. They are translated to TOP(1) filter in the SELECT clause. If a predicate is provided, the predicate is translated to WHERE clause. For example:

internal static void First(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

string first = source

.Select(product => product.Name)

.First() // Execute query.

.WriteLine();

// SELECT TOP(1) [product].[Name]

// FROM [Production].[Product] AS [product]

}

internal static void FirstOrDefault(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var firstOrDefault = source

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice })

.FirstOrDefault(product => product.ListPrice > 5000); // Execute query.

firstOrDefault?.Name.WriteLine();

// SELECT TOP(1) [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] > 5000.0

}

As discussed in LINQ to Objects, Single and SingleOrDefault are more strict. They are translated to TOP(2) filter, so that, if there are 0 or more than 1 results, InvalidOperationException is thrown. Similar to First and FirstOrDefault, if a predicate is provided, it is translated to WHERE clause:

internal static void Single(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var single = source

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice })

.Single(product => product.ListPrice < 50); // Execute query.

$"{single.Name}: {single.ListPrice}".WriteLine();

// SELECT TOP(2) [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] < 50.0

}

internal static void SingleOrDefault(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var singleOrDefault = source

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice })

.SingleOrDefault(product => product.ListPrice < 1); // Execute query.

singleOrDefault?.Name.WriteLine();

// SELECT TOP(2) [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] < 1.0

}

EF Core supports Last and LastOrDefault, locally. Again, if a predicate is provided, it is translated to WHERE clause:

internal static void Last(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

Product last = source.Last(); // Execute query.

// SELECT [p].[ProductID], [p].[ListPrice], [p].[Name], [p].[ProductSubcategoryID]

// FROM [Production].[Product] AS [p]

$"{last.Name}: {last.ListPrice}".WriteLine();

}

internal static void LastOrDefault(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

var lastOrDefault = source

.Select(product => new { Name = product.Name, ListPrice = product.ListPrice })

.LastOrDefault(product => product.ListPrice <= 0); // Execute query.

// SELECT [product].[Name], [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] <= 0.0

(lastOrDefault == null).WriteLine(); // True

}

The above examples can read many results from remote database to locally, and try to query the last result locally, which can cause performance issue.

Aggregation

Count/LongCount are translated to SQL aggregate functions COUNT/COUNT_BIG. if a is provided, it is translated to WHERE clause. The following examples query the System.Int32 count of categories, and the System.Int64 count of the products with list price greater than 0:

internal static void Count(AdventureWorks adventureWorks)

{

IQueryable<ProductCategory> source = adventureWorks.ProductCategories;

int count = source.Count().WriteLine(); // Execute query.

// SELECT COUNT(*)

// FROM [Production].[ProductCategory] AS [p]

}

internal static void LongCount(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

long longCount = source.LongCount(product => product.ListPrice > 0).WriteLine(); // Execute query.

// SELECT COUNT_BIG(*)

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] > 0.0

}

Max/Min/Sum/Average are translated to MAX/MIN/SUM/AVG functions. The following examples query the latest ModifiedDate of photos, the lowest list price of products, and the total cost of transactions, and the average ListPrice of products:

internal static void Max(AdventureWorks adventureWorks)

{

IQueryable<ProductPhoto> source = adventureWorks.ProductPhotos;

DateTime max = source.Select(photo => photo.ModifiedDate).Max().WriteLine(); // Execute query.

// SELECT MAX([photo].[ModifiedDate])

// FROM [Production].[ProductPhoto] AS [photo]

}

internal static void Min(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

decimal min = source.Min(product => product.ListPrice).WriteLine(); // Execute query.

// SELECT MIN([product].[ListPrice])

// FROM [Production].[Product] AS [product]

}

internal static void Sum(AdventureWorks adventureWorks)

{

IQueryable<TransactionHistory> source = adventureWorks.Transactions;

decimal sum = source.Sum(transaction => transaction.ActualCost).WriteLine(); // Execute query.

// SELECT SUM([transaction].[ActualCost])

// FROM [Production].[TransactionHistory] AS [transaction]

// WHERE [transaction].[TransactionType] IN (N'W', N'S', N'P')

}

internal static void Average(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

decimal average = source.Select(product => product.ListPrice).Average().WriteLine(); // Execute query.

// SELECT AVG([product].[ListPrice])

// FROM [Production].[Product] AS [product]

}

Quantifier

EF Core supports Contains for entity type, locally.

internal static void ContainsEntity(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

Product single = source.Single(product => product.ListPrice == 20.24M); // Execute query.

// SELECT TOP(2) [product].[ProductID], [product].[ListPrice], [product].[Name], [product].[ProductSubcategoryID]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] = 20.24

bool contains = source

.Where(product => product.ProductSubcategoryID == 7)

.Contains(single).WriteLine(); // Execute query.

// exec sp_executesql N'SELECT CASE

// WHEN @__p_0_ProductID IN (

// SELECT [product].[ProductID]

// FROM [Production].[Product] AS [product]

// WHERE [product].[ProductSubcategoryID] = 7

// )

// THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)

// END',N'@__p_0_ProductID int',@__p_0_ProductID=952

}

EF Core both support Contains for primitive types. In this case, Contains is translated to EXISTS predicate:

internal static void ContainsPrimitive(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

bool contains = source

.Select(product => product.ListPrice).Contains(100)

.WriteLine(); // Execute query.

// exec sp_executesql N'SELECT CASE

// WHEN @__p_0 IN (

// SELECT [product].[ListPrice]

// FROM [Production].[Product] AS [product]

// )

// THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)

// END',N'@__p_0 decimal(3,0)',@__p_0=100

}

Any is also translated to EXISTS. If predicate is provided, it is translated to WHERE clause:

internal static void Any(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

bool any = source.Any().WriteLine(); // Execute query.

// SELECT CASE

// WHEN EXISTS (

// SELECT 1

// FROM [Production].[Product] AS [p])

// THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)

// END

}

internal static void AnyWithPredicate(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

bool any = source.Any(product => product.ListPrice > 10).WriteLine(); // Execute query.

// SELECT CASE

// WHEN EXISTS (

// SELECT 1

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] > 10.0)

// THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)

// END

}

All is translated to NOT EXISTS, with the predicate translated to reverted condition in WHERE clause:

internal static void AllWithPredicate(AdventureWorks adventureWorks)

{

IQueryable<Product> source = adventureWorks.Products;

bool all = source.All(product => product.ListPrice > 10).WriteLine(); // Execute query.

// SELECT CASE

// WHEN NOT EXISTS (

// SELECT 1

// FROM [Production].[Product] AS [product]

// WHERE [product].[ListPrice] <= 10.0)

// THEN CAST(1 AS BIT) ELSE CAST(0 AS BIT)

// END

}

Summary

Text:

Entity Framework Core and LINQ to Entities in Depth (3) Logging and Tracing Queries

Sat, 05 Oct 2019 00:00:00 GMT

[LINQ via C# series]

[Entity Framework Core (EF Core) series]

[Entity Framework (EF) series]

As fore mentioned, LINQ to Entities queries are translated to database queries. To understand how EF Core work with databases, it is important to uncover the actual underlying operations to the SQL database, which can be traced or logged in C# application side and in SQL database.

Application side logging

EF Core follows the ASP.NET Core logging infrastructure. To log EF Core operations, a logger (implementing Microsoft.Extensions.Logging.ILogger) and a logger provider (implementing Microsoft.Extensions.Logging.ILoggerProvider) can be defined. The following is a simple example to simply trace everything:

public class TraceLogger : ILogger

private readonly string categoryName;

public TraceLogger(string categoryName) => this.categoryName = categoryName;

public bool IsEnabled(LogLevel logLevel) => true;

public void Log<TState>(

LogLevel logLevel,

EventId eventId,

TState state,

Exception exception,

Func<TState, Exception, string> formatter)

Trace.WriteLine($"{DateTime.Now.ToString("o")} {logLevel} {eventId.Id} {this.categoryName}");

Trace.WriteLine(formatter(state, exception));

public IDisposable BeginScope<TState>(TState state) => null;

public class TraceLoggerProvider : ILoggerProvider

public ILogger CreateLogger(string categoryName) => new TraceLogger(categoryName);

public void Dispose() { }

}

Now the logger provider can be hooked up with EF Core:

public partial class AdventureWorks

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)

LoggerFactory loggerFactory = new LoggerFactory();

loggerFactory.AddProvider(new TraceLoggerProvider());

optionsBuilder.UseLoggerFactory(loggerFactory);

}

The following is a simple example of LINQ to Entities query. It pulls all ProductCategory entities from AdventureWorks.ProductCategories data source:

internal static void Logger()

using (AdventureWorks adventureWorks = new AdventureWorks())

IQueryable<ProductCategory> source = adventureWorks.ProductCategories; // Define query.

source.WriteLines(category => category.Name); // Execute query.

// 2017-01-11T22:15:43.4625876-08:00 Debug 2 Microsoft.EntityFrameworkCore.Query.Internal.SqlServerQueryCompilationContextFactory

// Compiling query model:

// 'from ProductCategory <generated>_0 in DbSet<ProductCategory>

// select < generated>_0'

// 2017-01-11T22:15:43.4932882-08:00 Debug 3 Microsoft.EntityFrameworkCore.Query.Internal.SqlServerQueryCompilationContextFactory

// Optimized query model:

// 'from ProductCategory <generated>_0 in DbSet<ProductCategory>

// select < generated>_0'

// 2017-01-11T22:15:43.6179834-08:00 Debug 5 Microsoft.EntityFrameworkCore.Query.Internal.SqlServerQueryCompilationContextFactory

// TRACKED: True

// (QueryContext queryContext) => IEnumerable<ProductCategory> _ShapedQuery(

// queryContext: queryContext,

// shaperCommandContext: SelectExpression:

// SELECT [p].[ProductCategoryID], [p].[Name]

// FROM [Production].[ProductCategory] AS [p]

// ,

// shaper: UnbufferedEntityShaper<ProductCategory>

// )

// 2017-01-11T22:15:43.7272876-08:00 Debug 3 Microsoft.EntityFrameworkCore.Storage.Internal.SqlServerConnection

// Opening connection to database 'AdventureWorks' on server 'tcp:dixin.database.windows.net,1433'.

// 2017-01-11T22:15:44.1024201-08:00 Information 1 Microsoft.EntityFrameworkCore.Storage.IRelationalCommandBuilderFactory

// Executed DbCommand (66ms) [Parameters=[], CommandType='Text', CommandTimeout='30']

// SELECT [p].[ProductCategoryID], [p].[Name]

// FROM [Production].[ProductCategory] AS [p]

// 2017-01-11T22:15:44.1505353-08:00 Debug 4 Microsoft.EntityFrameworkCore.Storage.Internal.SqlServerConnection

// Closing connection to database 'AdventureWorks' on server 'tcp:dixin.database.windows.net,1433'.

}

The logs uncovers that a SELECT statement is executed in database to query all categories. The logs also uncovers how exactly EF Core execute the operation – it compiles LINQ to Entities query and generates SQL, then opens a connection to SQL database, execute the generated SQL in database, and close the connection. This mechanism is discussed in the query translation part.

EF Core also provides a TagWith query to annotate the translated database query:

internal static void TagWith(AdventureWorks adventureWorks)

IQueryable<ProductCategory> source = adventureWorks.ProductCategories

.TagWith("Query categories with id greater than 1.")

.Where(category => category.ProductCategoryID > 1); // Define query.

source.WriteLines(category => category.Name); // Execute query.

// -- Query categories with id greater than 1.

// SELECT [category].[ProductCategoryID], [category].[Name]

// FROM [Production].[ProductCategory] AS [category]

// WHERE [category].[ProductCategoryID]> 1

}

Database side tracing with Extended Events

SQL database provides variant mechanisms to collect the information of executed operations. Extended Events is such a feature available in all cloud and on-premise SQL database editions. For Windows, SQL Server Management Studio is a rich tools to setup and views the event tracing. And this can also be done from other platform. In any SQL tool (like mssql extension for Visual Studio Code, which works on Linux, Mac, and Windows), connect to the Azure SQL database (or SQL Server on-premise database), and execute the following SQL to create an Extended Events session called Queries:

CREATE EVENT SESSION [Queries] ON DATABASE -- ON SERVER for SQL Server on-premise database.

ADD EVENT sqlserver.begin_tran_completed(

ACTION(sqlserver.client_app_name, sqlserver.client_connection_id, sqlserver.client_hostname, sqlserver.client_pid, sqlserver.database_name, sqlserver.request_id, sqlserver.session_id, sqlserver.sql_text)),

ADD EVENT sqlserver.commit_tran_completed(

ACTION(sqlserver.client_app_name, sqlserver.client_connection_id, sqlserver.client_hostname, sqlserver.client_pid, sqlserver.database_name, sqlserver.request_id, sqlserver.session_id, sqlserver.sql_text)),

ADD EVENT sqlserver.error_reported(

ACTION(sqlserver.client_app_name, sqlserver.client_connection_id, sqlserver.client_hostname, sqlserver.client_pid, sqlserver.database_name, sqlserver.request_id, sqlserver.session_id, sqlserver.sql_text)),

ADD EVENT sqlserver.rollback_tran_completed(

ACTION(sqlserver.client_app_name, sqlserver.client_connection_id, sqlserver.client_hostname, sqlserver.client_pid, sqlserver.database_name, sqlserver.request_id, sqlserver.session_id, sqlserver.sql_text)),

ADD EVENT sqlserver.rpc_completed(

ACTION(sqlserver.client_app_name, sqlserver.client_connection_id, sqlserver.client_hostname, sqlserver.client_pid, sqlserver.database_name, sqlserver.request_id, sqlserver.session_id, sqlserver.sql_text)),

ADD EVENT sqlserver.sp_statement_completed(

ACTION(sqlserver.client_app_name, sqlserver.client_connection_id, sqlserver.client_hostname, sqlserver.client_pid, sqlserver.database_name, sqlserver.request_id, sqlserver.session_id, sqlserver.sql_text)),

ADD EVENT sqlserver.sql_batch_completed(

ACTION(sqlserver.client_app_name, sqlserver.client_connection_id, sqlserver.client_hostname, sqlserver.client_pid, sqlserver.database_name, sqlserver.request_id, sqlserver.session_id, sqlserver.sql_text)),

ADD EVENT sqlserver.sql_statement_completed(

ACTION(sqlserver.client_app_name, sqlserver.client_connection_id, sqlserver.client_hostname, sqlserver.client_pid, sqlserver.database_name, sqlserver.request_id, sqlserver.session_id, sqlserver.sql_text))

ADD TARGET package0.ring_buffer(SET max_events_limit = (100)) -- Most recent 100 events.

WITH (STARTUP_STATE = OFF);

It traces the transactions, SQL executions, and errors, etc. To start the session and collect events, execute the following SQL:

ALTER EVENT SESSION [Queries] ON DATABASE -- ON SERVER for SQL Server on-premise database.

STATE = START;

The collected events data is stored as XML, the following query formats the XML data to a statistics table, along with an event table which has the operations requested by .NET Core (or .NET Framework) application:

DECLARE @target_data XML =

(SELECT CONVERT(XML, [targets].[target_data])

FROM sys.dm_xe_database_session_targets AS [targets] -- sys.dm_xe_session_targets for SQL Server on-premise database.

INNER JOIN sys.dm_xe_database_sessions AS [sessions] -- sys.dm_xe_sessions for SQL Server on-premise database.

ON [sessions].[address] = [targets].[event_session_address]

WHERE [sessions].[name] = N'Queries');

SELECT

@target_data.value('(RingBufferTarget/@truncated)[1]', 'bigint') AS [truncated],

@target_data.value('(RingBufferTarget/@processingTime)[1]', 'bigint') AS [processingTime],

@target_data.value('(RingBufferTarget/@totalEventsProcessed)[1]', 'bigint') AS [totalEventsProcessed],

@target_data.value('(RingBufferTarget/@eventCount)[1]', 'bigint') AS [eventCount],

@target_data.value('(RingBufferTarget/@droppedCount)[1]', 'bigint') AS [droppedCount],

@target_data.value('(RingBufferTarget/@memoryUsed)[1]', 'bigint') AS [memoryUsed];

SELECT

[event].value('@timestamp[1]', 'datetime') AS [timestamp],

[event].value('(action[@name="client_hostname"]/value)[1]', 'nvarchar(MAX)') AS [client_hostname],

[event].value('(action[@name="client_pid"]/value)[1]', 'bigint') AS [client_pid],

[event].value('(action[@name="client_connection_id"]/value)[1]', 'uniqueidentifier') AS [client_connection_id],

[event].value('(action[@name="session_id"]/value)[1]', 'bigint') AS [session_id],

[event].value('(action[@name="request_id"]/value)[1]', 'bigint') AS [request_id],

[event].value('(action[@name="database_name"]/value)[1]', 'nvarchar(MAX)') AS [database_name],

[event].value('@name[1]', 'nvarchar(MAX)') AS [name],

[event].value('(data[@name="duration"]/value)[1]', 'bigint') AS [duration],

[event].value('(data[@name="result"]/text)[1]', 'nvarchar(MAX)') AS [result],

[event].value('(data[@name="row_count"]/value)[1]', 'bigint') AS [row_count],

[event].value('(data[@name="cpu_time"]/value)[1]', 'bigint') as [cpu_time],

[event].value('(data[@name="logical_reads"]/value)[1]', 'bigint') as [logical_reads],

[event].value('(data[@name="physical_reads"]/value)[1]', 'bigint') as [physical_reads],

[event].value('(data[@name="writes"]/value)[1]', 'bigint') as [writes],

[event].value('(action[@name="sql_text"]/value)[1]', 'nvarchar(MAX)') AS [sql_text],

[event].value('(data[@name="statement"]/value)[1]', 'nvarchar(MAX)') AS [statement],

[event].value('(data[@name="error_number"]/value)[1]', 'bigint') AS [error_number],

[event].value('(data[@name="message"]/value)[1]', 'nvarchar(MAX)') AS [message]

FROM @target_data.nodes('//RingBufferTarget/event') AS [Rows]([event])

WHERE [event].value('(action[@name="client_app_name"]/value)[1]', 'nvarchar(MAX)') = N'Core .Net SqlClient Data Provider' -- N'.Net SqlClient Data Provider' for .NET Framework.

ORDER BY [timestamp];

The following is an example of how the traced database operations look like:

Entity Framework Core and LINQ to Entities in Depth (2) Modeling Database: Object-Relational Mapping

Fri, 04 Oct 2019 00:00:00 GMT

[LINQ via C# series]

[Entity Framework Core (EF Core) series]

[Entity Framework (EF) series]

In LINQ to Entities, the queries are based on Object-relational mapping. .NET and SQL database and have 2 different data type systems. For example, .NET has System.Int64 and System.String, while SQL database has bigint and nvarchar; .NET has sequences and objects, while SQL database has tables and rows;, etc. Object-relational mapping is a popular technology to map and convert between application data objects and database relational data.

Data types

EF Core can map most SQL data types to .NET types:

<table border="1" cellpadding="0" cellspacing="0" class="MsoNormalTable" style="border: currentcolor; border-image: none; border-collapse: collapse; mso-border-alt: solid black .75pt; mso-yfti-tbllook: 1184;"><tbody><tr style="mso-yfti-irow: 0; mso-yfti-firstrow: yes;"><td style="padding: 0.75pt; border: 1pt solid black; border-image: none; mso-border-alt: solid black .75pt;">SQL type category</td><td style="border-width: 1pt 1pt 1pt medium; border-style: solid solid solid none; border-color: black black black currentcolor; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt;">SQL type</td><td style="border-width: 1pt 1pt 1pt medium; border-style: solid solid solid none; border-color: black black black currentcolor; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt;">.NET type</td><td style="border-width: 1pt 1pt 1pt medium; border-style: solid solid solid none; border-color: black black black currentcolor; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt;">C# primitive</td></tr><tr style="mso-yfti-irow: 1;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">Exact numeric</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">bit</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Boolean</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">bool</td></tr><tr style="mso-yfti-irow: 2;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">tinyint</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Byte</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">byte</td></tr><tr style="mso-yfti-irow: 3;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">smallint</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Int16</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">short</td></tr><tr style="mso-yfti-irow: 4;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">int</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Int32</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">int</td></tr><tr style="mso-yfti-irow: 5;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">bigint</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Int64</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">long</td></tr><tr style="mso-yfti-irow: 6;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">smallmoney, money, decimal, numeric</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Decimal</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">decimal</td></tr><tr style="mso-yfti-irow: 7;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">Approximate numeric</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">real</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Single</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">float</td></tr><tr style="mso-yfti-irow: 8;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">float</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Double</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">double</td></tr><tr style="mso-yfti-irow: 9;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">Character string</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">char, varchar, text</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.String</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">string</td></tr><tr style="mso-yfti-irow: 10;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">nchar, nvarchar, ntext</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.String</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">string</td></tr><tr style="mso-yfti-irow: 11;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">Binary string</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">binary, varbinary</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Byte[]</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">byte[]</td></tr><tr style="mso-yfti-irow: 12;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">image</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Byte[]</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">byte[]</td></tr><tr style="mso-yfti-irow: 13;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">rowversion (timestamp)</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Byte[]</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">byte[]</td></tr><tr style="mso-yfti-irow: 14;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">Date time</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">date</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.DateTime</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td></tr><tr style="mso-yfti-irow: 15;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">time</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.TimeSpan</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td></tr><tr style="mso-yfti-irow: 16;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">smalldatetime, datetime, datetime2</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.DateTime</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td></tr><tr style="mso-yfti-irow: 17;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">datetimeoffset</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.DateTimeOffset</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td></tr><tr style="mso-yfti-irow: 18;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">Other</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">hierarchyid</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">No built-in support</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td></tr><tr style="mso-yfti-irow: 19;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">xml</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.String</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">string</td></tr><tr style="mso-yfti-irow: 20;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">uniqueidentifier</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Guid</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td></tr><tr style="mso-yfti-irow: 21; mso-yfti-lastrow: yes;"><td style="border-width: medium 1pt 1pt; border-style: none solid solid; border-color: currentcolor black black; padding: 0.75pt; border-image: none; mso-border-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;"></td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">sql_variant</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">System.Object</td><td style="border-width: medium 1pt 1pt medium; border-style: none solid solid none; border-color: currentcolor black black currentcolor; padding: 0.75pt; mso-border-alt: solid black .75pt; mso-border-left-alt: solid black .75pt; mso-border-top-alt: solid black .75pt;">object</td></tr></tbody></table>

The mapping of spatial types are supported through NetTopologySuite, a free and open source library. For SQL database, just install the Microsoft.EntityFrameworkCore.SqlServer.NetTopologySuite NuGet package.

Database

A SQL database is mapped to a type derived from Microsoft.EntityFrameworkCore.DbContext:

public partial class AdventureWorks : DbContext { }

The following is the definition of DbContext:

namespace Microsoft.EntityFrameworkCore

public class DbContext : IDisposable, IInfrastructure<IServiceProvider>

public DbContext(DbContextOptions options);

public virtual ChangeTracker ChangeTracker { get; }

public virtual DatabaseFacade Database { get; }

public virtual void Dispose();

public virtual int SaveChanges();

public virtual DbSet<TEntity> Set<TEntity>() where TEntity : class;

protected internal virtual void OnModelCreating(ModelBuilder modelBuilder);

// Other members.

}

DbContext implements IDisposable. Generally, a database instance should be constructed and disposed for each unit of work - a collection of data operations that should succeed or fail as a unit:

internal static void Dispose()

using (AdventureWorks adventureWorks = new AdventureWorks())

// Unit of work.

}

EF Core also support DbContext pooling to improve performance. In the application or service, If DbContext is used through dependency injection, and it is no custom state (just like the above AdventureWorks type with no fields), then DbContext pooling can be enabled to reuse DbContext without disposing.

In EF Core, most of the object-relational mapping can be implemented declaratively, and the rest of the mapping can be implemented imperatively by overriding DbContext.OnModelCreating, which is automatically called by EF Core during initialization:

public partial class AdventureWorks

protected override void OnModelCreating(ModelBuilder modelBuilder)

base.OnModelCreating(modelBuilder);

MapCompositePrimaryKey(modelBuilder);

MapManyToMany(modelBuilder);

MapDiscriminator(modelBuilder);

}

The above MapCompositePrimaryKey, MapManyToMany, MapDiscriminator functions are implemented later in this chapter.

Connection resiliency and execution retry strategy

The connection to the mapped database can be specified from the constructor of DbContext:

public partial class AdventureWorks

public AdventureWorks(DbConnection connection = null)

: base(GetDbContextOptions(connection))

private static DbContextOptions GetDbContextOptions(

DbConnection connection = null) =>

new DbContextOptionsBuilder<AdventureWorks>()

.UseSqlServer(

connection: connection ??

new SqlConnection(ConnectionStrings.AdventureWorks),

sqlServerOptionsAction: options => options.EnableRetryOnFailure(

maxRetryCount: 5, maxRetryDelay: TimeSpan.FromSeconds(30),

errorNumbersToAdd: null))

.Options;

}

Here when database connection is not provided to the constructor, a new database connection is created with the previously defined connection string. Also, regarding the connection between application and SQL database may be interrupted over the network, EF Core supports connection resiliency for SQL database. This is very helpful for Azure SQL database deployed in the cloud instead of local network. In the above example, EF Core is specified to automatically retries up to 5 times with the retry interval of 30 seconds.

Tables

There are tens of tables in the AdventureWorks database, but don’t panic, this book only involves a few tables, and a few columns of these tables. In EF Core, a table definition can be mapped to an entity type definition, where each column is mapped to a entity property. For example, the AdventureWorks database has a Production.ProductCategory table. Its definition can be virtually viewed as:

CREATE SCHEMA [Production];

GO

CREATE TYPE [dbo].[Name] FROM nvarchar(50) NULL;

GO

CREATE TABLE [Production].[ProductCategory](

[ProductCategoryID] int IDENTITY(1,1) NOT NULL

CONSTRAINT [PK_ProductCategory_ProductCategoryID] PRIMARY KEY CLUSTERED,

[Name] [dbo].[Name] NOT NULL, -- nvarchar(50).

[rowguid] uniqueidentifier ROWGUIDCOL NOT NULL -- Ignored in mapping.

CONSTRAINT [DF_ProductCategory_rowguid] DEFAULT (NEWID()),

[ModifiedDate] datetime NOT NULL -- Ignored in mapping.

CONSTRAINT [DF_ProductCategory_ModifiedDate] DEFAULT (GETDATE()));

This table definition can be mapped to a ProductCategory entity definition:

public partial class AdventureWorks

public const string Production = nameof(Production); // Production schema.

[Table(nameof(ProductCategory), Schema = AdventureWorks.Production)]

public partial class ProductCategory

[Key]

[DatabaseGenerated(DatabaseGeneratedOption.Identity)]

public int ProductCategoryID { get; set; }

[MaxLength(50)]

[Required]

public string Name { get; set; }

// Other columns are ignored.

}

The [Table] attribute specifies the table name and schema. [Table] can be omitted when the table name is the same as the entity name, and the table is under the default dbo schema. In the table-entity mapping:

· The ProductCategoryID column of int type is mapped to a System.Int32 property with the same name. The [Key] attribute indicates it is a primary key. EF Core requires a table to have primary key to be mapped. [DatabaseGenerated] indicates it is an identity column, with value generated by database.

· The Name column is of dbo.Name type. which is actually nvarchar(50), so it is mapped to Name property of type System.String. The [MaxLength] attribute indicates the max length of the string value is 50. [Required] indicates it should not be null or empty string or whitespace string.

· The other columns rowguid and ModifiedDate are not mapped. They are not used in this book to keep the code examples simple.

At runtime, each row of Production.ProductCategory table is mapped to a ProductCategory entity instance. The entire table can be mapped to an IQueryable<T> data source, exposed as a property of the database mapping. EF Core provides DbSet<T>, which implements IQueryable<T>, to represent a table data source:

public partial class AdventureWorks

public DbSet<ProductCategory>ProductCategories { get; set; }

}

EF Core also supports immutable entity definition:

[Table(nameof(ProductCategory), Schema = AdventureWorks.Production)]

public partial class ProductCategory

public ProductCategory(int productCategoryID, string name) =>

(this.ProductCategoryID, this.Name) = (productCategoryID, name);

[Key]

[DatabaseGenerated(DatabaseGeneratedOption.Identity)]

public int ProductCategoryID { get; private set; }

[MaxLength(50)]

[Required]

public string Name { get; private set; }

}

This book defines all table mapping as mutable, since it is easier to update the entities and save back to database.

Relationships

In SQL database, tables can have foreign key relationships, including one-to-one, one-to-many, and many-to-many relationships.

One-to-one

The following Person.Person table and HumanResources.Employee table has a one-to-one relationship:

HumanResources.Employee table’s BusinessEntityID column is a foreign key that refers to Person.Person table’s primary key. Their definition can be virtually viewed as:

CREATE TABLE [Person].[Person](

[BusinessEntityID] int NOT NULL

CONSTRAINT [PK_Person_BusinessEntityID] PRIMARY KEY CLUSTERED,

[FirstName] [dbo].[Name] NOT NULL,

[LastName] [dbo].[Name] NOT NULL

/* Other columns. */);

GO

CREATE TABLE [HumanResources].[Employee](

[BusinessEntityID] int NOT NULL

CONSTRAINT [PK_Employee_BusinessEntityID] PRIMARY KEY CLUSTERED

CONSTRAINT [FK_Employee_Person_BusinessEntityID] FOREIGN KEY

REFERENCES [Person].[Person] ([BusinessEntityID]),

[JobTitle] nvarchar(50) NOT NULL,

[HireDate] date NOT NULL

/* Other columns. */);

So each row in HumanResources.Employee table refers to one row in Person.Person table (an employee must be a person). On the other hand, each row in Person.Person table can be referred by 0 or 1 row in HumanResources.Employee table (a person can be an employee, or not). This relationship can be represented by navigation property of entity type:

public partial class AdventureWorks

public const string Person = nameof(Person);

public const string HumanResources = nameof(HumanResources);

public DbSet<Person> People { get; set; }

public DbSet<Employee> Employees { get; set; }

[Table(nameof(Person), Schema = AdventureWorks.Person)]

public partial class Person

[Key]

public int BusinessEntityID { get; set; }

[Required]

[MaxLength(50)]

public string FirstName { get; set; }

[Required]

[MaxLength(50)]

public string LastName { get; set; }

public virtual Employee Employee { get; set; } // Reference navigation property.

[Table(nameof(Employee), Schema = AdventureWorks.HumanResources)]

public partial class Employee

[Key]

[ForeignKey(nameof(Person))]

public int BusinessEntityID { get; set; }

[Required]

[MaxLength(50)]

public string JobTitle { get; set; }

public DateTime HireDate { get; set; }

public virtual Person Person { get; set; } // Reference navigation property.

}

The [ForeignKey] attribute indicates Employee entity’s BusinessEntityID property is the foreign key for the relationship represented by navigation property. Here Person is called the primary entity, and Employee is called the dependent entity. Their navigation properties are called reference navigation properties, because each navigation property can refer to a single entity. The navigation property is designed to be virtual to enable proxy entity to implement lazy loading. Proxy entity and lazy loading is discussed in the query translation and data loading chapter.

One-to-many

The Production.ProductCategory and Production.ProductSubcategory tables have a one-to-many relationship, so are Production.ProductSubcategory and Production.Product:

Each row in Production.ProductCategory table can refer to many rows in Production.ProductSubcategory table (category can have many subcategories), and each row in Production.ProductSubcategory table can refer to many rows in Production.Product table (subcategory can have many products). Their definitions can be virtually viewed as:

CREATE TABLE [Production].[ProductSubcategory](

[ProductSubcategoryID] int IDENTITY(1,1) NOT NULL

CONSTRAINT [PK_ProductSubcategory_ProductSubcategoryID] PRIMARY KEY CLUSTERED,

[Name] [dbo].[Name] NOT NULL, -- nvarchar(50).

[ProductCategoryID] int NOT NULL

CONSTRAINT [FK_ProductSubcategory_ProductCategory_ProductCategoryID] FOREIGN KEY

REFERENCES [Production].[ProductCategory] ([ProductCategoryID]),

/* Other columns. */)

GO

CREATE TABLE [Production].[Product](

[ProductID] int IDENTITY(1,1) NOT NULL

CONSTRAINT [PK_Product_ProductID] PRIMARY KEY CLUSTERED,

[Name] [dbo].[Name] NOT NULL, -- nvarchar(50).

[ListPrice] money NOT NULL,

[ProductSubcategoryID] int NULL

CONSTRAINT [FK_Product_ProductSubcategory_ProductSubcategoryID] FOREIGN KEY

REFERENCES [Production].[ProductSubcategory] ([ProductSubcategoryID])

/* Other columns. */)

These one-to-many relationships can be represented by navigation property of type ICollection<T>:

public partial class ProductCategory

public virtual ICollection<ProductSubcategory> ProductSubcategories { get; set; } // Collection navigation property.

[Table(nameof(ProductSubcategory), Schema = AdventureWorks.Production)]

public partial class ProductSubcategory

[Key]

[DatabaseGenerated(DatabaseGeneratedOption.Identity)]

public int ProductSubcategoryID { get; set; }

[MaxLength(50)]

[Required]

public string Name { get; set; }

public int ProductCategoryID { get; set; }

public virtual ProductCategory ProductCategory { get; set; } // Reference navigation property.

public virtual ICollection<Product> Products { get; set; } // Collection navigation property.

[Table(nameof(Product), Schema = AdventureWorks.Production)]

public partial class Product

[Key]

[DatabaseGenerated(DatabaseGeneratedOption.Identity)]

public int ProductID { get; set; }

[MaxLength(50)]

[Required]

public string Name { get; set; }

public decimal ListPrice { get; set; }

public int? ProductSubcategoryID { get; set; }

public virtual ProductSubcategory ProductSubcategory { get; set; } // Reference navigation property.

}

Notice Production.Product table’s ProductSubcategoryID column is nullable, so it is mapped to a int? property. Here [ForeignKey] attribute is omitted, because each dependent entity’ foreign key is separated from its primary key, so the foreign key can be automatically discovered by EF Core.

Many-to-many

Production.Product and Production.ProductPhoto tables has many-to-many relationship.

This is implemented by 2 one-to-many relationships with another Production.ProductProductPhoto junction table. These tables’ definitions can be virtually viewed as:

CREATE TABLE [Production].[ProductPhoto](

[ProductPhotoID] int IDENTITY(1,1) NOT NULL

CONSTRAINT [PK_ProductPhoto_ProductPhotoID] PRIMARY KEY CLUSTERED,

[LargePhotoFileName] nvarchar(50) NULL,

[ModifiedDate] datetime NOT NULL

CONSTRAINT [DF_ProductPhoto_ModifiedDate] DEFAULT (GETDATE())

/* Other columns. */)

GO

CREATE TABLE [Production].[ProductProductPhoto](

[ProductID] int NOT NULL

CONSTRAINT [FK_ProductProductPhoto_Product_ProductID] FOREIGN KEY

REFERENCES [Production].[Product] ([ProductID]),

[ProductPhotoID] int NOT NULL

CONSTRAINT [FK_ProductProductPhoto_ProductPhoto_ProductPhotoID] FOREIGN KEY

REFERENCES [Production].[ProductPhoto] ([ProductPhotoID]),

CONSTRAINT [PK_ProductProductPhoto_ProductID_ProductPhotoID] PRIMARY KEY NONCLUSTERED ([ProductID], [ProductPhotoID])

/* Other columns. */)

So the many-to-many relationship can be mapped to 2 one-to-many relationships with the junction:

public partial class Product

public virtual ICollection<ProductProductPhoto> ProductProductPhotos { get; set; } // Collection navigation property.

[Table(nameof(ProductPhoto), Schema = AdventureWorks.Production)]

public partial class ProductPhoto

[Key]

[DatabaseGenerated(DatabaseGeneratedOption.Identity)]

public int ProductPhotoID { get; set; }

[MaxLength(50)]

public string LargePhotoFileName { get; set; }

[ConcurrencyCheck]

public DateTime ModifiedDate { get; set; }

public virtual ICollection<ProductProductPhoto> ProductProductPhotos { get; set; } // Collection navigation property.

[Table(nameof(ProductProductPhoto), Schema = AdventureWorks.Production)]

public partial class ProductProductPhoto

[Key]

[Column(Order = 0)]

public int ProductID { get; set; }

[Key]

[Column(Order = 1)]

public int ProductPhotoID { get; set; }

public virtual Product Product { get; set; } // Reference navigation property.

public virtual ProductPhoto ProductPhoto { get; set; } // Reference navigation property.

}

ProductPhoto.ModifiedDate has a [ConcurrencyCheck] attribute for concurrency conflict check, which is discussed in the data manipulation chapter. Production.ProductProductPhoto table has a composite primary key. As a junction table, each row in the table has a unique combination of ProductID and ProductPhotoID. EF Core requires additional initialization for composite primary key, which should be executed in DbContext’s OnModelCreating method:

public partial class AdventureWorks

private static void MapCompositePrimaryKey(ModelBuilder modelBuilder) // Called by OnModelCreating.

modelBuilder.Entity<ProductProductPhoto>()

.HasKey(productProductPhoto => new

ProductID = productProductPhoto.ProductID,

ProductPhotoID = productProductPhoto.ProductPhotoID

});

}

EF Core also requires additional initialization for many-to-many relationship represented by 2 one-to-many relationships, which should be executed in OnModelCreating as well:

public partial class AdventureWorks

private static void MapManyToMany(ModelBuilder modelBuilder) // Called by OnModelCreating.

modelBuilder.Entity<ProductProductPhoto>()

.HasOne(productProductPhoto => productProductPhoto.Product)

.WithMany(product => product.ProductProductPhotos)

.HasForeignKey(productProductPhoto => productProductPhoto.ProductID);

modelBuilder.Entity<ProductProductPhoto>()

.HasOne(productProductPhoto => productProductPhoto.ProductPhoto)

.WithMany(photo => photo.ProductProductPhotos)

.HasForeignKey(productProductPhoto => productProductPhoto.ProductPhotoID);

}

Finally, all the above tables can be exposed as properties of AdventureWorks:

public partial class AdventureWorks

public DbSet<Person> People { get; set; }

public DbSet<Employee> Employees { get; set; }

public DbSet<ProductSubcategory>ProductSubcategories { get; set; }

public DbSet<Product> Products { get; set; }

public DbSet<ProductPhoto> ProductPhotos { get; set; }

}

Inheritance

EF Core supports table per hierarchy (TPH) inheritance for entity types. With TPH, one table is mapped to many entity types in the inheritance hierarchy, so a discriminator column is needed to identify each specific row’s mapping entity. Take the Production.TransactionHistory table as example, its definition can be virtually viewed as:

CREATE TABLE [Production].[TransactionHistory](

[TransactionID] int IDENTITY(100000,1) NOT NULL

CONSTRAINT [PK_TransactionHistory_TransactionID] PRIMARY KEY CLUSTERED,

[ProductID] int NOT NULL

CONSTRAINT [FK_TransactionHistory_Product_ProductID] FOREIGN KEY

REFERENCES [Production].[Product] ([ProductID]),

[TransactionDate] datetime NOT NULL,

[TransactionType] nchar(1) NOT NULL

CONSTRAINT [CK_Product_Style]

CHECK (UPPER([TransactionType]) = N'P' OR UPPER([TransactionType]) = N'S' OR UPPER([TransactionType]) = N'W'),

[Quantity] int NOT NULL,

[ActualCost] money NOT NULL

/* Other columns. */);

Its TransactionType column allows value “P”, “S”, or “W” to indicate each row representing a purchase transaction, sales transaction, or work transaction. So the mapping entities’ hierarchy can be:

[Table(nameof(TransactionHistory), Schema = AdventureWorks.Production)]

public abstract class TransactionHistory

[Key]

public int TransactionID { get; set; }

public int ProductID { get; set; }

public DateTime TransactionDate { get; set; }

public int Quantity { get; set; }

public decimal ActualCost { get; set; }

public class PurchaseTransactionHistory : TransactionHistory { }

public class SalesTransactionHistory : TransactionHistory { }

public class WorkTransactionHistory : TransactionHistory { }

Then the discriminator must be specified when OnModelCreating is executed:

public enum TransactionType { P, S, W }

public partial class AdventureWorks

private static void MapDiscriminator(ModelBuilder modelBuilder) // Called by OnModelCreating.

modelBuilder.Entity<TransactionHistory>()

.HasDiscriminator<string>(nameof(TransactionType))

.HasValue<PurchaseTransactionHistory>(nameof(TransactionType.P))

.HasValue<SalesTransactionHistory>(nameof(TransactionType.S))

.HasValue<WorkTransactionHistory>(nameof(TransactionType.W));

}

Now these entities can all be exposed as data sources:

public partial class AdventureWorks

public DbSet<TransactionHistory>Transactions { get; set; }

public DbSet<PurchaseTransactionHistory> PurchaseTransactions { get; set; }

public DbSet<SalesTransactionHistory>SalesTransactions { get; set; }

public DbSet<WorkTransactionHistory>WorkTransactions { get; set; }

}

Views

A view can also be mapped as if it is a table, if the view has one or more columns which can be virtually viewed as primary key. Take the Production.vEmployee view as example, its definition can be virtually viewed as:

CREATE VIEW [HumanResources].[vEmployee]

AS

SELECT

e.[BusinessEntityID],

p.[FirstName],

p.[LastName],

e.[JobTitle]

-- Other columns.

FROM [HumanResources].[Employee] e

INNER JOIN [Person].[Person] p

ON p.[BusinessEntityID] = e.[BusinessEntityID]

/* Other tables. */;

The BusinessEntityID is unique and can be the virtual primary key. So it can be mapped as the following entity:

[Table(nameof(vEmployee), Schema = AdventureWorks.HumanResources)]

public class vEmployee

[Key]

public int BusinessEntityID { get; set; }

public string FirstName { get; set; }

public string LastName { get; set; }

public string JobTitle { get; set; }

}

And then expose as data source:

public partial class AdventureWorks

public DbSet<vEmployee> vEmployees { get; set; }

}

Summary

Text:

Entity Framework Core and LINQ to Entities in Depth (1) Remote Query

Tue, 01 Oct 2019 00:00:00 GMT

[LINQ via C# series]

[Entity Framework Core (EF Core) series]

[Entity Framework (EF) series]

Entity Framework Core

The previous chapters discussed LINQ to Objects, LINQ to XML, and Parallel LINQ. All of these LINQ technologies query local in-memory objects managed by .NET. This chapter discusses a different kind of LINQ technology, LINQ to Entities, which queries relational data managed by databases. LINQ to Entities was initially provided by Entity Framework (EF), a Microsoft library released since .NET Framework 3.5 Service Pack 1. Since 2016, Microsoft also released Entity Framework Core (EF Core), along with .NET Core. EF Core is based on .NET Standard, so it works cross-platform.

EF Core implements a provider model, so that LINQ to Entities can be implemented by different providers to work with different kinds of databases, including SQL Server (on-premise database) and Azure SQL Database (cloud database, aka SQL Azure), DB2, MySQL, Oracle, PostgreSQL, SQLLite, etc.

SQL database

To demonstrate LINQ to Entities queries and other database operations, this book uses the classic sample SQL database AdventureWorks provided by Microsoft as the data source, because this sample database has a very intuitive structure, it also works with Azure SQL Database and all SQL Server editions. The full sample database provided by Microsoft is relatively large, so a trimmed version is provided in the code samples repo of this book:

· The AdventureWorks.bacpac file is for Azure SQL Database

· The AdventureWorks_Data.mdf and AdventureWorks_Log.ldf files are for SQL Server

There are many free options to setup SQL database. To setup in the cloud, follow these steps:

Sign up Azure free trial program, or sign up Visual Studio Dev Essentials program, to get free Azure account and free credits.
Sign in to Azure portal, create a storage account, then create a container, and upload the above bacpac file into the container.
In Azure portal, create a SQL Database server, then add local IP address to the server’s firewall settings to enable access.
In Azure portal, import the uploaded bacpac file from the storage account to the server, and create a SQL database. There the many pricing tier options for the database creation, where the Basic tier starts from about $5 per month, which can be covered by the free credit.

As a alternative to cloud, SQL Server on premise can also be installed locally, then the above mdf and ldf files can be attached:

· On Windows, there are several free options to install SQL Server:

o SQL Server LocalDB: the easiest option, with no configuration required for setup.

o SQL Server Express Core

o SQL Server Express with Advanced Services

o SQL Server Developer Edition: free after signing up Visual Studio Dev Essentials program

o SQL Server Evaluation for the next version

· On Linux, SQL Server Express, Developer, and Evaluation editions are freely licensed.

· On Mac, SQL Server can be installed using a Windows/Linux virtual machine, or Docker

After setting up, tools can be optionally installed to connect to and manage the SQL database:

· On Windows, there are rich tools:

o SQL Server Data Tools for Visual Studio, a free Visual Studio extension enabling SQL database management inside Visual Studio

o SQL Server Management Tools, which includes SQL Server Management Studio (a free integration environment to manage SQL database), SQL Server Profiler (a free tracing tool for SQL Server on premise), and other tools.

· On Windows, Linux, and macOS:

o SQL Server (mssql) for Visual Studio Code, an extension for Visual Studio Code to execute SQL

o Azure Data Studio, a free cross-platform tool to manage data and edit query.

To connect to the sample database, its connection string can be saved in the configuration of application or service during development and test. For .NET Core, the connection string can be saved for the application as a JSON file, for example, as app.json file:

{

"ConnectionStrings": {

"AdventureWorks": "Server=tcp:dixin.database.windows.net,1433;Initial Catalog=AdventureWorks;Persist Security Info=False;User ID=***;Password=***;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;"

}

For .NET Framework, the connection string can be saved in the application’s app.config file:

<?xml version="1.0" encoding="utf-8"?>

<configuration>

<connectionStrings>

<add name="AdventureWorks" connectionString="Server=tcp:dixin.database.windows.net,1433;Initial Catalog=AdventureWorks;Persist Security Info=False;User ID=***;Password=***;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;" />

</connectionStrings>

</configuration>

Then the connection string can be loaded and used in C# code:

internal static class ConnectionStrings

internal static string AdventureWorks { get; } =

#if NETFX

ConfigurationManager.ConnectionStrings[nameof(AdventureWorks)].ConnectionString;

#else

new ConfigurationBuilder().AddJsonFile("App.json").Build()

.GetConnectionString(nameof(AdventureWorks));

#endif

}

The connection string for production should be protected with encryption or tools like Azure Key Vault configuration provider.

Remote query vs. local query

LINQ to Objects, Parallel LINQ query .NET objects in current .NET application’s local memory, these queries are called local queries. LINQ to XML queries XML data source, which are local .NET objects representing XML structures as well, so LINQ to XML queries are also local queries. As demonstrated at the beginning of this book, LINQ can also query data in other data domains, like tweets in Twitter, rows in database tables, etc. Apparently, these data source are not .NET objects directly available in local memory. These queries are called remote queries.

Remote LINQ (like LINQ to Entities) is provided as paraty of local LINQ (like LINQ to Objects). Since local data sources and local queries are represented by IEnumerable<T>, remote LINQ data sources (like a table in database) and remote queries (like a database query), are represented by System.Linq.IQueryable<T>:

namespace System.Linq

public interface IQueryable : IEnumerable

Expression Expression { get; }

Type ElementType { get; }

IQueryProvider Provider { get; }

public interface IOrderedQueryable : IQueryable, IEnumerable { }

public interface IQueryable<out T> : IEnumerable<T>, IEnumerable, IQueryable { }

public interface IOrderedQueryable<out T> : IQueryable<T>, IEnumerable<T>, IOrderedQueryable, IQueryable, IEnumerable { }

}

.NET Standard and Microsoft libraries provide many implementation of IEnumerable<T>, like T[] representing array, List<T> representing mutable list, Microsoft.Collections.Immutable.ImmutableList<T> representing immutable list, etc. EF Core also provides implementation of IQueryable<T>, including Microsoft.EntityFrameworkCore.DbSet<T> representing database table, Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable<T> representing database query, etc.

As the parity with System.Linq.Enumerable, System.Linq.Queryable static type provides the remote version of standard queries. For example, the following are the local and remote Where/Select/Concat/Cast queries side by side:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> predicate);

public static IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector);

public static IEnumerable<TSource> Concat<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second);

public static IEnumerable<TResult> Cast<TResult>(this IEnumerable source);

// Other members.

public static class Queryable

public static IQueryable<TSource> Where<TSource>(

this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate);

public static IQueryable<TResult> Select<TSource, TResult>(

this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector);

public static IQueryable<TSource> Concat<TSource>(

this IQueryable<TSource> source1, IEnumerable<TSource> source2);

public static IQueryable<TResult> Cast<TResult>(this IQueryable source);

// Other members.

}

When defining each standard query in remote LINQ, the generic source and generic output are represented by IQueryable<T> instead of IEnumerable<T>, and the non-generic source is represented by IQueryable instead of IEnumerable. The iteratee functions are replaced by expression trees. Similarly, the following are the ordering queries side by side, where the ordered source and ordered output are represented by IOrderedQueryable<T> instead of IOrderedEnumerable<T>:

namespace System.Linq

public static class Enumerable

public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> OrderByDescending<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> ThenBy<TSource, TKey>(

this IOrderedEnumerable<TSource>source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> ThenByDescending<TSource, TKey>(

this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static class Queryable

public static IOrderedQueryable<TSource> OrderBy<TSource, TKey>(

this IQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector);

public static IOrderedQueryable<TSource> OrderByDescending<TSource, TKey>(

this IQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector);

public static IOrderedQueryable<TSource> ThenBy<TSource, TKey>(

this IOrderedQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector);

public static IOrderedQueryable<TSource> ThenByDescending<TSource, TKey>(

this IOrderedQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector);

}

With this design, the fluent function chaining and the LINQ query expression pattern are automatically enabled for remote LINQ queries. It is the same syntax to write LINQ to Objects query and remote LINQ query.

Queryable does not provide the following queries:

· Empty/Range/Repeat: it does not make sense for .NET to locally generate a remote data source or remote query on the fly; the other generation query DefaultIfEmpty is available, because DefaultIfEmpty works with an existing IQueryable<T> source.

· AsEnumerable: Enumerable.AsEnumerable types any IEnumerable<T> source just as IEnumerable<T>. Since IQueryable<T> implements IEnumerable<T>, Enumerable.AsEnumerable also works for IQueryanle<T>.

· ToArray/ToDictionary/ToList/ToLookup: LINQ to Objects provides these colection queries to pull values from any IEnumerable<T> source and create local .NET collections. Since IQueryable<T> implements IEnumerable<T>, these queries provided by LINQ to Objects also works for IQueryanle<T>.

· Max/Min overloads for .NET primary types: these are specific types of local .NET application, not the remote data domain.

Queryable also provides an additional query AsQueryable, as the paraty with AsEnumerable. However, unlike AsSequential/AsParallel switching between sequential and parallel query, AsEnumerable/AsQueryable cannot freely switch between local and remote query. This query is discussed later.

Function vs. expression tree

Enumerable queries accept iteratee functions, and Queryable queries accept expression trees. As discussed in the lamda expression chapter, functions are executable .NET code, and expression trees are data structures representing the abstract syntax tree of functions, which can be translated to other domain-specific language. The lambda expression chapter also demonstrates compiling an arithmetic expression tree to CIL code at runtime, and executing it dynamically. The same approach can be used to translate arithmetic expression tree to SQL query, and execute it in a remote SQL database. The following function traverses an arithmetic expression tree with +, -, *, / operators, and compile it to a SQL SELECT statement with infix arithmetic expression:

internal static string InOrder(this LambdaExpression expression)

string VisitNode(Expression node)

switch (node.NodeType)

case ExpressionType.Constant when node is ConstantExpression constant:

return constant.Value.ToString();

case ExpressionType.Parameter when node is ParameterExpression parameter:

return $"@{parameter.Name}";

// In-order output: left child, current node, right child.

case ExpressionType.Add when node is BinaryExpression binary:

return $"({VisitNode(binary.Left)} + {VisitNode(binary.Right)})";

case ExpressionType.Subtract when node is BinaryExpression binary:

return $"({VisitNode(binary.Left)} - {VisitNode(binary.Right)})";

case ExpressionType.Multiply when node is BinaryExpression binary:

return $"({VisitNode(binary.Left)} * {VisitNode(binary.Right)})";

case ExpressionType.Divide when node is BinaryExpression binary:

return $"({VisitNode(binary.Left)} / {VisitNode(binary.Right)})";

default:

throw new ArgumentOutOfRangeException(nameof(expression));

return $"SELECT {VisitNode(expression.Body)};";

}

Here @ is prepended to each parameter name, which is the SQL syntax. The following code demonstrates the compilation:

internal static void Infix()

Expression<Func<double, double, double, double, double, double>> expression =

(a, b, c, d, e) => a + b - c * d / 2D + e * 3D;

string sql = expression.InOrder();

sql.WriteLine(); // SELECT (((@a + @b) - ((@c * @d) / 2)) + (@e * 3));

}

The following ExecuteSql function is defined to execute the compiled SQL statement with SQL parameters and SQL database connection string provided, and return the execution result from SQL database:

internal static double ExecuteSql(

string connection,

string sql,

IDictionary<string, double> parameters)

using (SqlConnection sqlConnection = new SqlConnection(connection))

using (SqlCommand sqlCommand = new SqlCommand(sql, sqlConnection))

sqlConnection.Open();

parameters.ForEach(parameter => sqlCommand.Parameters.AddWithValue(parameter.Key, parameter.Value));

return (double)sqlCommand.ExecuteScalar();

}

And the following TranslateToSql function is defined to wrap the entire work. It accept an arithmetic expression tree, call the above InOrder to compile it to SQL, then emit a dynamic function, which extracts the parameters and calls above ExecuteScalar function to execute the SQL:

public static TDelegate TranslateToSql<TDelegate>(

this Expression<TDelegate> expression, string connection)

DynamicMethod dynamicMethod = new DynamicMethod(

string.Empty,

expression.ReturnType,

expression.Parameters.Select(parameter => parameter.Type).ToArray(),

MethodBase.GetCurrentMethod().Module);

EmitCil(dynamicMethod.GetILGenerator(), expression.InOrder());

return (TDelegate)(object)dynamicMethod.CreateDelegate(typeof(TDelegate));

void EmitCil(ILGenerator generator, string sql)

// Dictionary<string, double> dictionary = new Dictionary<string, double>();

generator.DeclareLocal(typeof(Dictionary<string, double>));

generator.Emit(

OpCodes.Newobj,

typeof(Dictionary<string, double>).GetConstructor(Array.Empty<Type>()));

generator.Emit(OpCodes.Stloc_0);

for (int index = 0; index < expression.Parameters.Count; index++)

// dictionary.Add($"@{expression.Parameters[i].Name}", args[i]);

generator.Emit(OpCodes.Ldloc_0); // dictionary.

generator.Emit(OpCodes.Ldstr, $"@{expression.Parameters[index].Name}");

generator.Emit(OpCodes.Ldarg_S, index);

generator.Emit(

OpCodes.Callvirt,

typeof(Dictionary<string, double>).GetMethod(

nameof(Dictionary<string, double>.Add),

BindingFlags.Instance | BindingFlags.Public | BindingFlags.InvokeMethod));

// ExecuteSql(connection, expression, dictionary);

generator.Emit(OpCodes.Ldstr, connection);

generator.Emit(OpCodes.Ldstr, sql);

generator.Emit(OpCodes.Ldloc_0);

generator.Emit(

OpCodes.Call,

new Func<string, string, IDictionary<string, double>, double>(ExecuteSql).Method);

generator.Emit(OpCodes.Ret); // Returns the result.

}

As fore mentioned, .NET built-in Expression<TDelegate>.Compile method compiles expression tree to CIL, and emits a function to execute the CIL locally with current .NET application process. In contrast, here TranslateToSql compiles the arithmetic expression tree to SQL query, and emits a function to execute the SQL in a specified remote SQL database:

internal static void TranslateAndExecute()

Expression<Func<double, double, double, double, double, double>> expression =

(a, b, c, d, e) => a + b - c * d / 2D + e * 3D;

Func<double, double, double, double, double, double> local = expression.Compile();

local(1, 2, 3, 4, 5).WriteLine(); // 12

Func<double, double, double, double, double, double> remote = expression.TranslateToSql(ConnectionStrings.AdventureWorks);

remote(1, 2, 3, 4, 5).WriteLine(); // 12

}

Parallel LINQ in Depth (4) Performance

Tue, 24 Sep 2019 00:00:00 GMT

[LINQ via C# series]

[Parallel LINQ in Depth series]

The purpose of PLINQ is to utilize multiple CPUs for better performance than LINQ to Objects However, PLINQ can also introduces performance overhead, like source partitioning and result merging. There are many aspects that impact PLINQ query performance.

Sequential query vs. parallel query

To compare the performance of sequential and parallel query, take OrderBy query as example. PLINQ’s OrderBy requires partitioning the source, as well as buffering and merging the results. The following function compares the query execution duration of sequential OrderBy and parallel OrderBy:

internal static void OrderByTest(

Func<int, int> keySelector, int sourceCount, int testRepeatCount)

int[] source = EnumerableX

.RandomInt32(min: int.MinValue, max: int.MaxValue, count: sourceCount)

.ToArray();

Stopwatch stopwatch = Stopwatch.StartNew();

Enumerable.Range(0, testRepeatCount).ForEach(_ =>

int[] sequentialResults = source.OrderBy(keySelector).ToArray();

});

stopwatch.Stop();

$"Sequential:{stopwatch.ElapsedMilliseconds}".WriteLine();

stopwatch.Restart();

Enumerable.Range(0, testRepeatCount).ForEach(_ =>

int[] parallel1Results = source.AsParallel().OrderBy(keySelector).ToArray();

});

stopwatch.Stop();

$"Parallel:{stopwatch.ElapsedMilliseconds}".WriteLine();

}

It calls the RandomInt32 query, which is defined in the LINQ to Objects custom queries chapter, to generate an array of random int values with the specified length. Then it executes the sequential and parallel OrderBy queries repeatedly for the specified times, so that the total execution time can be controlled in a reasonable range, the following code calls the OrderByTest function compares the sequential/parallel OrderBy execution on arrays of small/medium/large size, with the same simple key selector:

internal static void OrderByTestForSourceCount()

OrderByTest(keySelector: value => value, sourceCount: 5, testRepeatCount: 10_000);

// Sequential:11 Parallel:1422

OrderByTest(keySelector: value => value, sourceCount: 5_000, testRepeatCount: 100);

// Sequential:114 Parallel:107

OrderByTest(keySelector: value => value, sourceCount: 500_000, testRepeatCount: 100);

// Sequential:18210 Parallel:8204

}

The following code compares the sequential/parallel OrderBy execution on arrays of the same size, with different key selector of light/medium/heavy workload:

internal static void OrderByTestForKeySelector()

OrderByTest(

keySelector: value => value + ComputingWorkload(baseIteration: 1),

sourceCount: Environment.ProcessorCount, testRepeatCount: 100_000);

// Sequential:37 Parallel:2218

OrderByTest(

keySelector: value => value + ComputingWorkload(baseIteration: 10_000),

sourceCount: Environment.ProcessorCount, testRepeatCount: 1_000);

// Sequential:115 Parallel:125

OrderByTest(

keySelector: value => value + ComputingWorkload(baseIteration: 100_000),

sourceCount: Environment.ProcessorCount, testRepeatCount: 100);

// Sequential:1240 Parallel:555

}

It turns out PLINQ has better performance than LINQ to Objects with larger source and expensive iteratee function, which has a better chance to offset the overhead of partitioning and buffering/merging.

CPU bound operation vs. I/O bound operation

So far, all the examples are CPU bound operations. In most cases, PLINQ by default takes the logic processor count as the degree of parallelism. This makes sense for CPU bound operations, but may be not ideal for I/O bound operations. For example, when downloading files from Internet with parallel threads, it could be nice if the worker thread count can be controlled accurately disregarding the CPU core count. The following ForceParallel extension method can be implementation for this purpose:

internal static void ForceParallel<TSource>(

this IEnumerable<TSource> source, Action<TSource> iteratee, int degreeOfParallelism)

if (degreeOfParallelism <= 1)

throw new ArgumentOutOfRangeException(nameof(degreeOfParallelism));

IList<IEnumerator<TSource>> partitions = Partitioner

.Create(source, EnumerablePartitionerOptions.NoBuffering) // Stripped partitioning.

.GetPartitions(degreeOfParallelism);

ConcurrentBag<Exception> exceptions = new ConcurrentBag<Exception>();

void IteratePartition(IEnumerator<TSource> partition)

try

using (partition)

while (partition.MoveNext())

iteratee(partition.Current);

catch (Exception exception)

exceptions.Add(exception);

Thread[] threads = partitions

.Skip(1)

.Select(partition => new Thread(() => IteratePartition(partition)))

.ToArray();

threads.ForEach(thread => thread.Start());

IteratePartition(partitions[0]);

threads.ForEach(thread => thread.Join());

if (!exceptions.IsEmpty)

throw new AggregateException(exceptions);

}

It calls Partitioner.Create with EnumerablePartitionerOptions.NoBuffering, to enable stripped partitioning for better load balance. It then calls the created partitioner to create the specified number of partitions, and uses current thread and additional threads to simultaneously pull each partition and call the iterate function.

To demonstrate the I/O bound operation, the following function first visualizes sequential download, then visualizes parallel download with PLINQ, and finally visualizes parallel download with above ForceParallel function. Again, assuming a quad core CPU, the degree of parallelism is specified as 10, which is higher than the core count:

internal static void DownloadTest(string[] uris)

byte[] Download(string uri)

using (WebClient webClient = new WebClient())

return webClient.DownloadData(uri);

uris.Visualize(EnumerableEx.ForEach, uri => Download(uri).Length.WriteLine());

const int DegreeOfParallelism = 10;

uris.AsParallel()

.WithDegreeOfParallelism(DegreeOfParallelism)

.Visualize(ParallelEnumerable.ForAll, uri => Download(uri).Length.WriteLine());

uris.Visualize(

query: (source, iteratee) => source.ForceParallel(iteratee, DegreeOfParallelism),

iteratee: uri => Download(uri).Length.WriteLine());

}

The following code queries some thumbnail picture file URIs from the Flickr RSS feed with LINQ to XML, then pass the URIs to above function to visualize the download:

internal static void RunDownloadTestWithSmallFiles()

string[] smallThumbnailUris = XDocument

.Load("https://www.flickr.com/services/feeds/photos_public.gne?id=64715861@N07&format=rss2")

.Descendants((XNamespace)"http://search.yahoo.com/mrss/" + "thumbnail")

.Attributes("url")

.Select(uri => (string)uri)

.ToArray();

DownloadTest(smallThumbnailUris);

}

Here sequential download takes longer time, as expected. The PLINQ query is specified with a max degree of parallelism 10, but it decides to utilize 5 threads. ForceParallel starts 10 threads exactly as specified, and its execution time is about half of PLINQ.

The following code queries for the same Flickr RSS feed, but for large picture file URIs, and visualize the download:

internal static void RunDownloadTestWithLargeFiles()

string[] largePictureUris = XDocument

.Load("https://www.flickr.com/services/feeds/photos_public.gne?id=64715861@N07&format=rss2")

.Descendants((XNamespace)"http://search.yahoo.com/mrss/" + "content")

.Attributes("url")

.Select(uri => (string)uri)

.ToArray();

DownloadTest(largePictureUris);

}

This time PLINQ still utilizes 5 threads from the beginning, then decides to start 2 more threads a while later. ForceParallel simply start 10 threads since the beginning. However, the duration of sequential download, PLINQ download, and ForceParallel download are about the same. This is because when downloading larger files, the network bandwidth is fully occupied and becomes the performance bottleneck, so the degree of parallelism does not make much difference.

Factors to impact performance

This part and the previous parts have demonstrated many aspects that can have performance impact for PLINQ, and here is a summary:

· The partitioning strategy can impact performance, because different partitioning algorithms introduce different synchronization and load balance.

· The 2 execution modes, Default (sequential or parallel) and ForceParallel, can result different performance

· The degree of parallelism can impact performance, when degree of parallelism is set to 1, PLINQ works like sequential LINQ to Object.

· The merge option can also impact performance, smaller buffer size can have the early value results available faster, but can also make the query execute longer.

· The order preservation can impact the performance, query as unordered can have better performance, but can lead to incorrect results.

· The source size can impact performance, for source with smaller size, the overhead of parallelization can be more significant, and result even lower performance than sequential query.

· The iteratee function provided to query can impact performance, more expensive iteratee functions can have better performance with parallel queries.

· The type of operation can impact performance, utilize more CPU cores can improve the performance of compute bound operation, but I/O bound operations can also depend on the I/O hardware.

In the real world, the performance of each PLINQ query has to be measured and optimized accordingly.

Summary

PLINQ’s query execution performance is impacted by many aspects. First PLINQ partitions source for parallel query. PLINQ implements range partitioning, chuck partitioning, hash partitioning, and stripped partitioning. These partitioning algorithms require different synchronization, and result different load balance among the query threads to impact the overall performance. .NET Standard also provides APIs to define custom static, dynamic, and orderable partitioners. To utilize multi-processor and offset the overhead of partitioning and merging, PLINQ can have better performance with larger size source and more expensive iteratee function. PLINQ’s query performance also should be optimized according to the type of operation.

Parallel LINQ in Depth (3) Query Methods (Operators)

Mon, 23 Sep 2019 00:00:00 GMT

[LINQ via C# series]

[Parallel LINQ in Depth series]

Most of the PLINQ standard queries are the parities with LINQ to Objects standard queries, with the same syntax and functionality. For the additional queries in PLINQ, AsParallel, AsSequential and ForAll has been discussed. The rest of this chapter discusses the other additional queries, overloads, and the ordering relevant queries that have different behaviour from LINQ to Objects.

Query settings

PLINQ provides a few queries to configure the current cancellation token, degree of parallelism, execution mode, and merge options.

Cancellation

PLINQ query execution can be cancelled by providing a System.Threading.CancellationToken instance:

public static ParallelQuery<TSource> WithCancellation<TSource>(

this ParallelQuery<TSource> source, CancellationToken cancellationToken);

A CancellationToken instance can be created with System.Threading.CancellationTokenSource:

internal static void Cancel()

using (CancellationTokenSource cancellationTokenSource =

new CancellationTokenSource(delay: TimeSpan.FromSeconds(1)))

CancellationToken cancellationToken = cancellationTokenSource.Token;

try

ParallelEnumerable.Range(0, Environment.ProcessorCount * 10)

.WithCancellation(cancellationToken)

.Select(value => ComputingWorkload(value))

.WiteLines();

catch (OperationCanceledException exception)

exception.WriteLine();

// The query has been canceled via the token supplied to WithCancellation.

}

If the query executes for longer than 1 second, it is signalled to cancel, and throws an OperationCanceledException.

Degree of parallelism

WithDegreeOfParallelism specifies the maximum number of concurrent executing tasks:

public static ParallelQuery<TSource> WithDegreeOfParallelism<TSource>(

this ParallelQuery<TSource> source, int degreeOfParallelism);

For example:

internal static void DegreeOfParallelism()

int maxConcurrency = Environment.ProcessorCount * 10;

ParallelEnumerable

.Range(0, maxConcurrency)

.WithDegreeOfParallelism(maxConcurrency)

.Visualize(ParallelEnumerable.Select, value => value + ComputingWorkload())

.WriteLines();

}

WithDegreeOfParallelism accepts any int value from 1 to 512 (System.Linq.Parallel.Scheduling’s MAX_SUPPORTED_DOP constant field). At runtime, the actual query execution thread count is less than or equal to the specified maximum count. In the above example, WithDegreeOfParallelism is called with 40. However, when executing above query on a quad core CPU, the visualization shows PLINQ only utilizes 6 threads.

If WithDegreeOfParallelism is not called, the default degree of parallelism is the minimum value of current device’s processor count and 512:

namespace System.Linq.Parallel

internal static class Scheduling

internal const int MAX_SUPPORTED_DOP = 512;

internal static int DefaultDegreeOfParallelism = Math.Min(Environment.ProcessorCount, MAX_SUPPORTED_DOP);

internal static int GetDefaultDegreeOfParallelism() => DefaultDegreeOfParallelism;

}

Execution mode

WithExecutionMode specifies whether the query is allowed to execute sequentially or not:

public static ParallelQuery<TSource> WithExecutionMode<TSource>(

this ParallelQuery<TSource> source, ParallelExecutionMode executionMode);

ParallelExecutionMode is an enumeration type with 2 members. The Default mode of PLINQ means PLINQ can detect the composition of the query and possibly decide to execute the query sequentially; And ForceParallelism requires the query to execute in parallel. For example:

internal static void ExecutionMode()

int count = Environment.ProcessorCount * 2;

using (Markers.EnterSpan(-2, nameof(ParallelExecutionMode.Default)))

int result = ParallelEnumerable

.Range(0, count)

.SelectMany((value, index) => EnumerableEx.Return(value))

.Visualize(ParallelEnumerable.Select, value => value + ComputingWorkload())

.ElementAt(count - 1);

using (Markers.EnterSpan(-3, nameof(ParallelExecutionMode.ForceParallelism)))

int result = ParallelEnumerable

.Range(0, count)

.WithExecutionMode(ParallelExecutionMode.ForceParallelism)

.SelectMany((value, index) => EnumerableEx.Return(value))

.Visualize(ParallelEnumerable.Select, value => value + ComputingWorkload())

.ElementAt(count - 1);

}

When PLINQ execute the above query composed with indexed SelectMany and ElementAt, in the default mode, it is the same sequential execution as LINQ to Objects. To have parallel query execution, the ForceParallelism mode has to be specified.

Merging

PLINQ can partition the source values so that the query can be executed in parallel. To merge the results, WithMergeOptions can be used to suggest the strategy:

public static ParallelQuery<TSource> WithMergeOptions<TSource>(

this ParallelQuery<TSource> source, ParallelMergeOptions mergeOptions);

ParallelMergeOptions is an enumeration with 4 members. NotBuffered means when each result value is available, it is yielded immediately without being buffered., which is similar to lazy evaluation in LINQ to Objects; FullyBuffered means all results are stored in the fully sized buffer, then, they are yielded, which is similar to eager evaluation in LINQ to Objects; AutoBuffered is between NotBuffered and FullyBuffered, means the buffer size is determined by PLINQ, some results are stored in the auto sized buffer, and ones the buffer is full, the results are yielded; And Default is the same as AutoBuffered. The following code demonstrates the difference of these options:

internal static void MergeForSelect()

int count = 10;

Stopwatch stopwatch = Stopwatch.StartNew();

ParallelQuery<int>notBuffered = ParallelEnumerable.Range(0, count)

.WithMergeOptions(ParallelMergeOptions.NotBuffered)

.Select(value => value + ComputingWorkload());

notBuffered.ForEach(value => $"{value}:{stopwatch.ElapsedMilliseconds}".WriteLine());

// 0:217 3:283 6:363 8:462 1:521 4:612 7:629 9:637 2:660 5:695

stopwatch.Restart();

ParallelQuery<int>autoBuffered = ParallelEnumerable.Range(0, count)

.WithMergeOptions(ParallelMergeOptions.AutoBuffered)

.Select(value => value + ComputingWorkload());

autoBuffered.ForEach(value => $"{value}:{stopwatch.ElapsedMilliseconds}".WriteLine());

// 6:459 8:493 7:498 9:506 0:648 1:654 2:656 3:684 4:686 5:688

stopwatch.Restart();

ParallelQuery<int>fullyBuffered = ParallelEnumerable.Range(0, count)

.WithMergeOptions(ParallelMergeOptions.FullyBuffered)

.Select(value => value + ComputingWorkload());

fullyBuffered.ForEach(value => $"{value}:{stopwatch.ElapsedMilliseconds}".WriteLine());

// 0:584 1:589 2:618 3:627 4:629 5:632 6:634 7:636 8:638 9:641

}

For above Select query execution, if NotBuffered is specified, the first result value is yielded faster; if FullyBuffered is specified, the last result value is yielded faster; if AutoBuffered is specified, the behaviour is between NotBuffered and FullyBuffered. Also, since FullyBuffered buffers all results, it can preserve their order, while NotBuffered and AutoBuffered cannot. When pulling the results, ForEach is used here instead of ForAll to demonstrate that ParallelMergeOptions also impact the ordering of PLINQ query. When everything is fully buffered, PLINQ can have the ability to merge everything as ordered.

WithMergeOptions just provides a suggestion to PLINQ, so PLINQ can still make its own decision. For example, ForAll is NotBuffered, and yields whatever is pulled. If above queries are executed with ForAll instead of ForEach, all queries’ results become unordered. Another example is, OrderBy has to evaluate all source values, fully buffer them, then sort them:

internal static void MergeForOrderBy()

int count = Environment.ProcessorCount * 2;

Stopwatch stopwatch = Stopwatch.StartNew();

ParallelEnumerable.Range(0, count)

.WithMergeOptions(ParallelMergeOptions.NotBuffered)

.Select(value => ComputingWorkload(value))

.ForEach(value => $"{value}:{stopwatch.ElapsedMilliseconds}".WriteLine());

// 0:132 2:273 1:315 4:460 3:579 6:611 5:890 7:1103

stopwatch.Restart();

ParallelEnumerable.Range(0, count)

.WithMergeOptions(ParallelMergeOptions.NotBuffered)

.Select(value => ComputingWorkload(value))

.OrderBy(value => value) // Eager evaluation.

.ForEach(value => $"{value}:{stopwatch.ElapsedMilliseconds}".WriteLine());

// 0:998 1:999 2:999 3:1000 4:1000 5:1000 6:1001 7:1001

stopwatch.Restart();

ParallelEnumerable.Range(0, count)

.WithMergeOptions(ParallelMergeOptions.FullyBuffered)

.Select(value => ComputingWorkload(value))

.OrderBy(value => value) // Eager evaluation.

.ForEach(value => $"{value}:{stopwatch.ElapsedMilliseconds}".WriteLine());

// 0:984 1:985 2:985 3:986 4:987 5:987 6:988 7:989

}

So OrderBy ignores the suggested ParallelMergeOptions and always fully buffer the results.

Ordering

In PLINQ, it is more complex to control the order of values than in sequential LINQ to Objects. Apparently, the order of values may not be persisted when they are not sequentially processed. As demonstrated above, WithMergeOptions is one way to impact the order of query results, where ParallelMergeOptions.FullyBuffered can be specified to preserve the order. PLINQ also provides other APIs to control the order.

Preserving the order

AsOrdered query can be called to specify the order in the source should be preserved for its subsequent queries:

public static ParallelQuery<TSource> AsOrdered<TSource>(

this ParallelQuery<TSource> source);

AsOrdered can only be called on the ParallelQuery<T> instance which is the output of ParallelEnumerable.AsParallel, ParallelEnumerable.Range, or ParallelEnumerable.Repeat. It throws InvalidOperationException for ParallelQuery<T> instance output by any other queries.

internal static void AsOrdered()

ParallelEnumerable

.Range(0, Environment.ProcessorCount * 2)

.Select(value => value + ComputingWorkload())

.ForEach(value => value.WriteLine()); // 3 1 2 0 4 5 6 7

ParallelEnumerable

.Range(0, Environment.ProcessorCount * 2)

.AsOrdered()

.Select(value => value + ComputingWorkload())

.ForEach(value => value.WriteLine()); // 0 1 2 3 4 5 6 7

}

Here ForEach is used instead of ForAll again to demonstrate the order of query results. In contrast, AsUnordered is provided to ignore the order in the source for its subsequent queries:

public static ParallelQuery<TSource> AsUnordered<TSource>(

this ParallelQuery<TSource> source);

Ignoring the order may improve the query performance. Take GroupBy as example, it can run faster if the PLINQ query is explicitly specified to be unordered. The following example uses a tuple of string and int to represent a product’s name and weight, and group many products by their weight:

internal static void AsUnordered()

Random random = new Random();

(string Name, int Weight)[] products = ParallelEnumerable

.Range(0, Environment.ProcessorCount * 10_000)

.Select(_ => (Name: Guid.NewGuid().ToString(), Weight: random.Next(1, 10)))

.ToArray();

Stopwatch stopwatch = Stopwatch.StartNew();

products

.AsParallel()

.GroupBy(model => model.Weight)

.ForEach(group => group.Key.WriteLine());

stopwatch.Stop();

stopwatch.ElapsedMilliseconds.WriteLine(); // 800.

stopwatch.Restart();

products

.AsParallel()

.AsUnordered()

.GroupBy(model => model.Weight)

.ForEach(group => group.Key.WriteLine());

stopwatch.Stop();

stopwatch.ElapsedMilliseconds.WriteLine(); // 103.

}

When ordering queries OrderBy, OrderByDescending, ThenBy, ThenByDescending and Reverse) are called, the order is introduced and preserved in the subsequent queries:

internal static void OrderBy()

ParallelEnumerable

.Range(0, Environment.ProcessorCount * 2)

.Select(value => value) // Order is not preserved.

.ForEach(value => value.WriteLine()); // 3 1 2 0 4 5 6 7

ParallelEnumerable

.Range(0, Environment.ProcessorCount * 2)

.Select(value => value) // Order is not preserved.

.OrderBy(value => value) // Order is preserved.

.Select(value => value) // Order is preserved.

.ForEach(value => value.WriteLine()); // 0 1 2 3 4 5 6 7

}

Order and correctness

In PLINQ, many queries are order sensitive. If the source is unordered, the following queries output indeterministic results:

· Sequence queries:

o Reverse: does nothing

o SequenceEqual: compares values in arbitrary order

o Skip: skips arbitrary values

o SkipWhile: skips arbitrary values

o Take: takes arbitrary values

o TakeWhile: takes arbitrary values with the predicate

o Zip: zips unordered values

· Value queries:

o ElementAt: returns arbitrary value

o ElementAtOrDefault: returns arbitrary value or default

o First: returns arbitrary value

o FirstOrDefault: returns arbitrary value or default

o Last: returns arbitrary value

o LastOrDefault: returns arbitrary value or default

These queries must be used with ordered source to have the correct query results.

And, once again, ForAll pulls values and calls the specified function in parallel, and does not maintain the order as well.

PLINQ also provides ordered partitioner for order preservation. The partitioner and ordered partitioner are discussed in the next chapter.

Aggregation

In PLINQ, Aggregate query requires the provided accumulator functions to be both commutative and associative.

Commutativity, associativity and correctness

Assume func is a function that accepts 2 parameters and returns a result, if func(a, b) ≡ func(b, a), then func is commutative; if func(func(a, b), c) ≡ func(a, func(b, c)), then func is associative. For example:

internal static void CommutativeAssociative()

Func<int, int, int> func1 = (a, b) => a + b;

(func1(1, 2) == func1(2, 1)).WriteLine(); // True, commutative

(func1(func1(1, 2), 3) == func1(1, func1(2, 3))).WriteLine(); // True, associative.

Func<int, int, int> func2 = (a, b) => a * b + 1;

(func2(1, 2) == func2(2, 1)).WriteLine(); // True, commutative

(func2(func2(1, 2), 3) == func2(1, func2(2, 3))).WriteLine(); // False, not associative.

Func<int, int, int> func3 = (a, b) => a;

(func3(1, 2) == func3(2, 1)).WriteLine(); // False, not commutative

(func3(func3(1, 2), 3) == func3(1, func3(2, 3))).WriteLine(); // True, associative.

Func<int, int, int> func4 = (a, b) => a - b;

(func4(1, 2) == func4(2, 1)).WriteLine(); // False, not commutative

(func4(func4(1, 2), 3) == func4(1, func4(2, 3))).WriteLine(); // False, not associative.

}

To demonstrate how parallel aggregation is impacted by commutativity and associativity, it can be compared with sequential aggregation:

internal static void AggregateCorrectness()

int count = Environment.ProcessorCount * 2;

int sequentialAdd = Enumerable.Range(0, count).Aggregate((a, b) => a + b);

sequentialAdd.WriteLine(); // 28

int parallelAdd = ParallelEnumerable.Range(0, count).Aggregate((a, b) => a + b);

parallelAdd.WriteLine(); // 28

int sequentialSubtract = Enumerable.Range(0, count).Aggregate((a, b) => a - b);

sequentialSubtract.WriteLine(); // -28

int parallelSubtract = ParallelEnumerable.Range(0, count).Aggregate((a, b) => a - b);

parallelSubtract.WriteLine(); // 2

}

Apparently, parallelSubtract has incorrect result value, because the function provided to Aggregate is neither commutative nor associative. The following code visualizes the aggregation:

internal static void VisualizeAggregate()

int count = Environment.ProcessorCount * 2;

using (Markers.EnterSpan(-1, "Sequential subtract"))

MarkerSeries markerSeries = Markers.CreateMarkerSeries("Sequential subtract");

int sequentialSubtract = Enumerable.Range(0, count).Aggregate((a, b) =>

using (markerSeries.EnterSpan(Thread.CurrentThread.ManagedThreadId, $"{a}, {b} => {a - b}"))

return a - b + ComputingWorkload();

});

using (Markers.EnterSpan(-2, "Parallel subtract"))

MarkerSeries markerSeries = Markers.CreateMarkerSeries("Parallel subtract");

int parallelSubtract = ParallelEnumerable.Range(0, count).Aggregate((a, b) =>

using (markerSeries.EnterSpan(Thread.CurrentThread.ManagedThreadId, $"{a}, {b} => {a - b}"))

return a - b + ComputingWorkload();

});

}

The sequential aggregation has the expected process:

The parallel aggregation has different behaviours:

It follows the pattern of parallel query methods. It first partitions the data. On this quad core CPU, it splits the 8 source values into 4 partitions, (0, 1), (2, 3), (4, 5), (6, 7). Then it execute the provided function for each parallel in parallel, the 4 partitions’ result values are –1, –1, –1, –1. And finally it merges the 4 result values with the provided function, so the final aggregation result is 2. This demonstrates that the accumulator function must be commutative and associative for the parallel aggregation.

Merging

PLINQ provides 2 additional Aggregate overloads, where the seed for each partition is specified with either a value or a value factory function:

public static TResult Aggregate<TSource, TAccumulate, TResult>(

this ParallelQuery<TSource> source,

TAccumulate seed,

Func<TAccumulate, TSource, TAccumulate>updateAccumulatorFunc,

Func<TAccumulate, TAccumulate, TAccumulate>combineAccumulatorsFunc,

Func<TAccumulate, TResult> resultSelector);

public static TResult Aggregate<TSource, TAccumulate, TResult>(

this ParallelQuery<TSource> source,

Func<TAccumulate>seedFactory,

Func<TAccumulate, TSource, TAccumulate>updateAccumulatorFunc,

Func<TAccumulate, TAccumulate, TAccumulate>combineAccumulatorsFunc,

Func<TAccumulate, TResult> resultSelector);

They also both accept 2 accumulator functions. First, updateAccumulatorFunc can be read as “source value accumulator”, it accumulates the values within each partition to a partition result. Then, combineAccumulatorsFunc can be read as “partition result accumulator”, it accumulates all partitions’ results to a single final result. The following example calculates the sum of squares:

internal static void MergeForAggregate()

int count = Environment.ProcessorCount * 2;

int parallelSumOfSquares1 = ParallelEnumerable

.Range(0, count)

.Aggregate(

seed: 0, // Seed for each partition.

updateAccumulatorFunc: (accumulation, value) => accumulation + value * value, // Source value accumulator for each partition's result.

combineAccumulatorsFunc: (accumulation, partition) => accumulation + partition, // Partition result accumulator for final result.

resultSelector: result => result);

parallelSumOfSquares1.WriteLine(); // 140

int parallelSumOfSquares2 = ParallelEnumerable

.Range(0, count)

.Aggregate(

seedFactory: () => 0, // Seed factory for each partition.

updateAccumulatorFunc: (accumulation, value) => accumulation + value * value, // Source value accumulator for each partition's result.

combineAccumulatorsFunc: (accumulation, partition) => accumulation + partition, // Partition result accumulator for final result.

resultSelector: result => result);

parallelSumOfSquares2.WriteLine(); // 140

}

In the parallel aggregation, first the sum of squares is calculated for each partition. Then all partitions’ results are summed up to the final result.

Summary

PLINQ is a parity with LINQ to Objects that supports parallel execution. The query execution can be rendered as intuitive charts with Microsoft’s Concurrency Visualizer tool. PLINQ provides parallel standard queries as parities with LINQ to Objects’ sequential standard queries. PLINQ also provides additional queries for conversion between sequential query and parallel query, configuring parallel query, etc. Some PLINQ queries require the source to be ordered to have accurate query results, and the Aggregate query requires the accumulator function to be commutative and associative to have accurate query result.

Parallel LINQ in Depth (2) Partitioning

Sun, 22 Sep 2019 00:00:00 GMT

[LINQ via C# series]

[Parallel LINQ in Depth series]

The previous chapter discussed what is PLINQ and how to use PLINQ. This chapter looks into PLINQ’s internals and execution, including data processing and query performance.

Internal partitioning and load balancing

To execute query with multithreading, PLINQ must split the data source’s values for those query threads as the first step. Internally, PLINQ has 4 different data partitioning algorithms – range partitioning, chunk partitioning, strip partitioning, and hash partitioning. These partitioning algorithms lead to different load balancing among the multiple query threads, and impact the overall performance.

Range partitioning

Range partitioning works with indexed source with a known length, such as T [] arrays with an indexer and Length property, and IList<T> lists with an indexer and Count property. Assume on a quad core CPU, there are 12 values in the source array, by default PLINQ splits these 12 values (at indexes 0, 1, 2, …, 11) into 4 partition A, B, C, D as the following:

Index: 0 1 2 3 4 5 6 7 8 9 10 11

Partition: A A A B B B C C C D D D

If there are 13 source values, there are partitioned as: AAAA, BBB, CCC, DDD; 14 values are partitioned as AAAA, BBBB, CCC, DDD; 15 values are partitioned as AAAA, BBBB, CCCC, DDD; 16 values are partitioned as AAAA, BBBB, CCCC, DDDD; and so on.

With the Visualize and ComputingWorkload functions defined in the previous chapter, the following code can visualize how an array is partitioned by range of index:

internal static void RangePartitioningForArray()

int[] array = Enumerable.Range(0, Environment.ProcessorCount * 4).ToArray();

array.AsParallel()

.Visualize(ParallelEnumerable.Select, value => ComputingWorkload(value))

.WriteLines();

}

Execute the above code with Concurrency Visualizer for Visual Studio, the following chart is rendered:

Here the timespan of processing value 12 is longer than the timespan of processing value 15, because CPU was fully utilized at the beginning. Regarding there are also other processes and threads running on the device, when processing value 12, the query thread cannot ideally utilize 25% of CPU (100% of one core). In range partitioning, since each value of the source goes to a deterministic partition based on the value’s index, synchronization is not required for multiple query threads to simultaneously pull values from a shared source. This algorithm does not consider the actual work of the query threads, so it does not balance the load very well. For example, the partition (0, 1, 2, 3) is quickly processed by a query thread, then that thread becomes idle and just waits for other threads to be done with other partitions.

Chunk partitioning

Chunk partitioning can be used for sequence without index, where each thread pulls a chunk of values at a time. The chunk size starts from 1, and increases to 2, 4, 8, 16, …. Initially the chunk size is 1, each thread repeatedly pulls N chunks; Then the chunk size increases to 2, and each thread repeatedly pulls N chunks again; Then the chunk size increase to 4, and each thread repeatedly pulls another N chunk again; and so on. For a sequence, the chunk repeat count N is implemented as 8. Assume a quad core CPU, PLINQ split values in source into 4 partitions A, B, C, D by default, then the partitioning for source values is: ABCD, ABCD, …, AABBCCDD, AABBCCDD, …, AAAABBBBCCCCDDDD, AAAABBBBCCCCDDDD, ..., assuming each value’s processing cost the same time. The following code visualizes the chunk partitioning for a sequence without index:

internal static void ChunkPartitioningForSequence()

const int ChunkRepeatCount = 8;

IEnumerable<int> sequence = Enumerable.Range(

0, (1 + 2) * ChunkRepeatCount * Environment.ProcessorCount + 4);

sequence.AsParallel()

.Visualize(ParallelEnumerable.Select, value => value + ComputingWorkload())

.WriteLines();

}

When executing this query on a quad core CPU, for each query thread, the first 8 chunks have 1 value in each chunk, the next 8 chunks have 2 continuous values in each chunk, the last chunk has 4 continuous values, and they are all partitioned to one thread:

With chuck partitioning, synchronization is required for multiple query threads to access the shared source, so that each chunk of values is exclusively pulled by one thread. Internally, PLINQ utilizes C# lock statement with a synchronization object to synchronize the query threads. This approach can balance the load at chunk level, which has an incremental size.

When chunk partitioning is used for partitioner, the chunk repeat count N is implemented as 3. The easiest way to create a partitioner is to call Partitioner.Create with a sequence:

internal static void ChunkPartitioningForPartitioner()

const int ChunkRepeatCount = 3;

Partitioner<int> partitioner = Partitioner.Create(

Enumerable.Range(0, (1 + 2) * ChunkRepeatCount * Environment.ProcessorCount + 4));

partitioner.AsParallel()

.Visualize(ParallelEnumerable.Select, value => value + ComputingWorkload())

.WriteLines();

}

Here Partitioner.Create has a Partitioner<T> output. The Partitioner<TSource> type is the contract to implement partitioning, which is discussed later in this chapter. Then the ParallelEnumerable.AsParallel overload for Partitioner<T> can be called:

public static ParallelQuery<TSource> AsParallel<TSource>(

this Partitioner<TSource> source);

From there the PLINQ queries can be used subsequently. With a smaller repeat count, the chunks are more intuitive:

Hash partitioning

When PLINQ query needs to compare and group values in the source, like GroupBy, Join, GroupJoin, etc., it partitions the values based on hash code. To demonstrate this behaviour, a data structure with a custom hash algorithm can be defined:

internal readonly struct Data

internal Data(int value) => this.Value = value;

internal int Value { get; }

public override int GetHashCode() => this.Value % Environment.ProcessorCount;

public override string ToString() => this.Value.ToString(); // For span label.

}

It just wraps an int value, but only produces 4 different hash code on a quad core CPU. The following code visualize how GroupBy query executes its elementSelector function:

internal static void HashPartitioningForGroupBy()

IEnumerable<Data> sequence = new int[] { 0, 1, 2, 2, 2, 2, 3, 4, 5, 6, 10 }

.Select(value => new Data(value));

sequence.AsParallel()

.Visualize(

(source, elementSelector) => source.GroupBy(

keySelector: data => data, // Key's GetHashCode is called.

elementSelector: elementSelector),

data => ComputingWorkload(data.Value).ToString()) // elementSelector.

.WriteLines(group => string.Join(", ", group));

// Equivalent to:

// MarkerSeries markerSeries = Markers.CreateMarkerSeries("Parallel");

// source.AsParallel()

// .GroupBy(

// keySelector: data => data,

// elementSelector: data =>

// {

// using (markerSeries.EnterSpan(Thread.CurrentThread.ManagedThreadId, data.ToString()))

// {

// return ComputingWorkload(data.Value);

// }

// })

// .WriteLines(group => string.Join(", ", group));

}

Here GroupBy uses Data instances as the keys, it internally calls each Data instance’s GetHashCode method, and uses the output hash codes for equality comparison and grouping, then it processes the Data instances group by group with multiple query threads. As a result, Data instances with the same hash code is partitioned together and processed by the same query thread. Apparently, hash partitioning balances the load at group level. The synchronization work is required when pulling each group exclusively.

Similarly, the following example visualizes how Join query executes its resultSelector function:

internal static void HashPartitioningForJoin()

IEnumerable<Data> outerSource = new int[] { 0, 1, 2, 2, 2, 2, 3, 6 }.Select(value => new Data(value));

IEnumerable<Data> innerSource = new int[] { 4, 5, 6, 7 }.Select(value => new Data(value));

outerSource.AsParallel()

.Visualize(

(source, resultSelector) => source

.Join(

inner: innerSource.AsParallel(),

outerKeySelector: data => data, // Key's GetHashCode is called.

innerKeySelector: data => data, // Key's GetHashCode is called.

resultSelector: (outerData, innerData) => resultSelector(outerData)),

data => ComputingWorkload(data.Value)) // resultSelector.

.WriteLines();

}

Again, Data instances with the same hash code are partitioned together and processed by the same query thread:

Stripped partitioning

Stripped partitioning can work with source with or without index. In this algorithm, each PLINQ query thread just pulls one value from the source each time. when the thread finishes processing that value, it pulls another one value again, until the source has no value available. Still assume a quad core CPU, and assume it costs the same time for to process each value, then the partitioning result is:

Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ...

Partition: A B C D A B C D A B C D A B C D ...

Partitioner.Create has an overload for sequence source to create partitioner that implements stripped partitioning:

internal static void StrippedPartitioningForPartitionerWithSequence()

Partitioner<int> partitioner = Partitioner.Create(

Enumerable.Range(0, Environment.ProcessorCount * 10),

EnumerablePartitionerOptions.NoBuffering);

partitioner.AsParallel()

.Visualize(ParallelEnumerable.Select, value => ComputingWorkload())

.WriteLines();

}

Patitioner.Create’s EnumerablePartitionerOptions parameter has 2 members, None and NoBuffering. If None is specified, a partitioner is created to implement chunk partitioning; If NoBuffering is specified, a partitioner is created to implement stripped partitioning. The above code renders the following chart:

Partitioner.Create also provides similar overloads for array and list. Take the previous array as example:

internal static void StrippedPartitioningForPartitionerWithArray()

int[] array = Enumerable.Range(0, Environment.ProcessorCount * 4).ToArray();

Partitioner<int> partitioner = Partitioner.Create(array, loadBalance: true);

partitioner.AsParallel()

.Visualize(ParallelEnumerable.Select, value => ComputingWorkload(value))

.WriteLines();

}

Partitioner.Create’s loadBalance parameter is a bool value. If false is specified, a partitioner is created to implement range partitioning; If true is specified, a partitioner is created to implement stripped partitioning. The above code renders the following chart:

Implementing custom partitioner

.NET Standard also provides APIs to implement custom partitioning. The contract is the System.Collections.Partitioner<TSource> abstract class:

namespace System.Collections.Concurrent

public abstract class Partitioner<TSource>

protected Partitioner() { }

public virtual bool SupportsDynamicPartitions => false;

public abstract IList<IEnumerator<TSource>> GetPartitions(int partitionCount);

public virtual IEnumerable<TSource> GetDynamicPartitions() =>

throw new NotSupportedException("Dynamic partitions are not supported by this partitioner.");

}

Static partitioner

A partitioner’s SupportsDynamicPartitions property has a bool output. When it is false, the partitioner is a static partitioner, which means its GetPartitions method is available. To call GetPartitions, the partition count must be specified at the beginning, so the partition count is static and cannot be changed once partitioning is started. The output of GetPartitions method is a list of iterators, where each iterator is used to yield the values of a partition. This design of having multiple IEnumerator<T> iterators to share one IEnumerable<T> sequence, is the same idea as the EnumerableEx.Share and IBuffer<T> from Ix library discussed in the Ix chapter. So a simple static partitioner can be implemented as a wrapper of IBuffer<T> created by Share:

internal class StaticPartitioner<TSource> : Partitioner<TSource>

protected readonly IBuffer<TSource> Buffer;

internal StaticPartitioner(IEnumerable<TSource> source) => this.Buffer = source.Share();

public override IList<IEnumerator<TSource>> GetPartitions(int partitionCount)

if (partitionCount <= 0)

throw new ArgumentOutOfRangeException(nameof(partitionCount));

return Enumerable

.Range(0, partitionCount)

.Select(_ => this.Buffer.GetEnumerator())

.ToArray();

}

As demonstrated above, AsParallel can be called with partitioner:

internal static void QueryStaticPartitioner()

IEnumerable<int>source = Enumerable.Range(0, Environment.ProcessorCount * 4);

new StaticPartitioner<int>(source).AsParallel()

.Visualize(ParallelEnumerable.Select, value => ComputingWorkload(value))

.WriteLines();

}

The output IBuffer<T> of EnumerableEx.Share implements stripped partitioning. Similar to PLINQ, it internally also utilizes C# lock statement with a synchronization object to make sure each value is exclusively pulled by one thread. The above code renders the following chart:

Dynamic partitioner

When the output of SupportsDynamicPartitions property is true, the partitioner is a dynamic partitioner. Besides GetPartitions that splits source into specified number of partitions, dynamic partitioner’s GetDynamicPartitions is also available to split source into arbitrary number of partitions. The output of GetDynamicPartitions is a IEnumerable<T> sequence, whose GetEnumerator method can be called to output a IEnumerator<T> iterator that represents a partition. After partitioning is started, the output IEnumerable<T> sequence’s GetEnumerator method can be called again for arbitrary times, so the caller can have dynamic number of partitions. This scenario is still supported by IBuffer<T> from EnumerableEx.Share:

internal class DynamicPartitioner<TSource> : StaticPartitioner<TSource>

internal DynamicPartitioner(IEnumerable<TSource> source) : base(source) { }

public override bool SupportsDynamicPartitions => true;

public override IEnumerable<TSource> GetDynamicPartitions() => this.Buffer;

}

Besides PLINQ queries, dynamic partitioner can also be used with System.Threading.Tasks.Parallel’s ForEach function:

namespace System.Threading.Tasks

public static class Parallel

public static ParallelLoopResult ForEach<TSource>(Partitioner<TSource> source, Action<TSource> body);

}

Parallel.ForEach first checks SupportsDynamicPartitions. If it gets false, it throws an InvalidOperationException: The Partitioner used here must support dynamic partitioning; If it gets true, it then calls GetDynamicPartitions to partition the values and call the specified iteratee function in parallel for each partition:

internal static void QueryDynamicPartitioner()

IEnumerable<int>source = Enumerable.Range(0, Environment.ProcessorCount * 4);

Parallel.ForEach(

new DynamicPartitioner<int>(source), value => ComputingWorkload(value));

}

Parallel.ForEach has another overload accepting an IEnumerable<T> sequence, which is more commonly used:

public static ParallelLoopResult ForEach<TSource>(

IEnumerable<TSource> source, Action<TSource> body);

Internally, it calls the fore mentioned Partitioner.Create to create a dynamic partitioner from the source sequence.

Orderable partitioner

.NET also provides APIs for partitioning with order control. The contract is the System.Collections.OrderablePartitioner<TSource> abstract class, which is a subtype of Partitioner<TSource>. The following are the members of OrderablePartitioner<TSource>:

namespace System.Collections.Concurrent

public abstract class OrderablePartitioner<TSource> : Partitioner<TSource>

protected OrderablePartitioner(bool keysOrderedInEachPartition, bool keysOrderedAcrossPartitions, bool keysNormalized)

this.KeysOrderedInEachPartition = keysOrderedInEachPartition;

this.KeysOrderedAcrossPartitions = keysOrderedAcrossPartitions;

this.KeysNormalized = keysNormalized;

public bool KeysNormalized { get; }

public bool KeysOrderedInEachPartition { get; }

public bool KeysOrderedAcrossPartitions { get; }

public abstract IList<IEnumerator<KeyValuePair<long, TSource>>>GetOrderablePartitions(int partitionCount);

public virtual IEnumerable<KeyValuePair<long, TSource>>GetOrderableDynamicPartitions() =>

throw new NotSupportedException("Dynamic partitions are not supported by this partitioner.");

}

Instead of providing partitions of values, orderable partitioner provides partitions of key-value pairs, where key is the index of the value. Its GetOrderablePartitions method is the orderable parity with Partitioner<TSource>.GetPartitions, which gives a static count of partitions represented by iterators of index-value pairs; Its GetOrderableDynamicPartitions method is the orderable parity with Partitioner<TSource>.GetDynamicPartitions, which gives a sequence, where GetEnumerator can be called arbitrary times to get dynamic count of partitions; Its KeysNormalized property outputs a bool value to indicate whether the indexes increase from 0; Its KeysOrderedInEachPartition property indicates whether inside each partition, the indexes always increase, so that a later value’s index is always greater than an former value’s index; And its KeysOrderedAcrossPartitions property indicates whether indexes increase partition by partition, so that a later partition’s indexes are all greater than an former partition’s indexes. Once again, it is easy to implement orderable partitioner with EnumerableEx.Share and IBuffer<T>:

internal class OrderableDynamicPartitioner<TSource> : OrderablePartitioner<TSource>

private readonly IBuffer<KeyValuePair<long, TSource>> buffer;

internal OrderableDynamicPartitioner(IEnumerable<TSource> source)

: base(keysOrderedInEachPartition: true, keysOrderedAcrossPartitions: true, keysNormalized: true)

long index = -1;

this.buffer = source

.Select(value => new KeyValuePair<long, TSource>(Interlocked.Increment(ref index), value))

.Share();

public override bool SupportsDynamicPartitions => true;

public override IList<IEnumerator<KeyValuePair<long, TSource>>>GetOrderablePartitions(

int partitionCount) => Enumerable

.Range(0, partitionCount)

.Select(_ => this.buffer.GetEnumerator())

.ToArray();

public override IEnumerable<KeyValuePair<long, TSource>>GetOrderableDynamicPartitions() => this.buffer;

}

Once orderable partitioner is converted with AsParallel, AsOrdered can be used to preserve the order:

internal static partial class Partitioning

internal static void QueryOrderablePartitioner()

int[] source = Enumerable.Range(0, Environment.ProcessorCount * 2).ToArray();

new OrderableDynamicPartitioner<int>(source)

.AsParallel()

.Select(value => value + ComputingWorkload())

.WriteLines(); // 1 0 5 3 4 6 2 7

new OrderableDynamicPartitioner<int>(source)

.AsParallel()

.AsOrdered()

.Select(value => value + ComputingWorkload())

.WriteLines(); // 0 1 2 3 4 5 6 7

new DynamicPartitioner<int>(source)

.AsParallel()

.AsOrdered()

.Select(value => value + ComputingWorkload())

.WriteLines();

// InvalidOperationException: AsOrdered may not be used with a partitioner that is not orderable.

}

Parallel LINQ in Depth (1) Local Parallel Query and Visualization

Fri, 20 Sep 2019 00:00:00 GMT

[LINQ via C# series]

[Parallel LINQ in Depth series]

LINQ to Objects and LINQ to XML queries are designed to work sequentially, and do not involve multi-threading, concurrency, or parallel computing. To scale LINQ query in multi-processor environment, .NET Standard provides parallel version of LINQ to Objects, called Parallel LINQ or PLINQ.

Parallel LINQ query

Parallel LINQ (to Objects) APIs are provided as a parity with (sequential) LINQ to Objects APIs:

As the parity with System.Linq.Enumerable, System.Linq.ParallelEnumerable static type provides the parallel version of standard queries. For example, the following is the comparison of the Range/Repeat generation queries’ sequential and parallel versions:

namespace System.Linq

public static class Enumerable

public static IEnumerable<int> Range(int start, int count);

public static IEnumerable<TResult> Repeat<TResult>(TResult element, int count);

// Other members.

public static class ParallelEnumerable

public static ParallelQuery<int> Range(int start, int count);

public static ParallelQuery<TResult> Repeat<TResult>(TResult element, int count);

// Other members.

}

And the following are the sequential and parallel Where/Select/Concat/Cast queries side by side:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> predicate);

public static IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector);

public static IEnumerable<TSource> Concat<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second);

public static IEnumerable<TResult> Cast<TResult>(this IEnumerable source);

public static class ParallelEnumerable

public static ParallelQuery<TSource> Where<TSource>(

this ParallelQuery<TSource> source, Func<TSource, bool> predicate);

public static ParallelQuery<TResult> Select<TSource, TResult>(

this ParallelQuery<TSource> source, Func<TSource, TResult> selector);

public static ParallelQuery<TSource> Concat<TSource>(

this ParallelQuery<TSource> first, ParallelQuery<TSource> second);

public static ParallelQuery<TResult> Cast<TResult>(this ParallelQuery source);

}

When defining each standard query in PLINQ, the generic source and generic output are represented by ParallelQuery<T> instead of IEnumerable<T>, and the non-generic source is represented by ParallelQuery instead of IEnumerable. The other parameter types remain the same. Similarly, the following are the ordering queries side by side, where the ordered source and ordered output are represented by OrderedParallelQuery<T> instead of IOrderedEnumerable<T>:

namespace System.Linq

public static class Enumerable

public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> OrderByDescending<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> ThenBy<TSource, TKey>(

this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> ThenByDescending<TSource, TKey>(

this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static class ParallelEnumerable

public static OrderedParallelQuery<TSource> OrderBy<TSource, TKey>(

this ParallelQuery<TSource> source, Func<TSource, TKey> keySelector);

public static OrderedParallelQuery<TSource> OrderByDescending<TSource, TKey>(

this ParallelQuery<TSource> source, Func<TSource, TKey> keySelector);

public static OrderedParallelQuery<TSource> ThenBy<TSource, TKey>(

this OrderedParallelQuery<TSource> source, Func<TSource, TKey> keySelector);

public static OrderedParallelQuery<TSource> ThenByDescending<TSource, TKey>(

this OrderedParallelQuery<TSource> source, Func<TSource, TKey> keySelector);

}

With this design, the fluent function chaining and the LINQ query expression pattern are automatically enabled for PLINQ queries. It is the same syntax to write LINQ to Objects query and PLINQ query.

Besides the parities with Enumerable queries, ParallelEnumerable also provides additional queries and additional overloads for Aggregate query:

· Sequence queries

o Conversion: AsParallel, AsSequential

o Query settings: WithCancellation, WithDegreeOfParallelism, WithExecutionMode, WithMergeOptions

o Ordering: AsOrdered, AsUnordered

· Value queries

o Aggregation: Aggregate

· Void queries

o Iteration: ForAll

Parallel query vs. sequential query

A ParallelQuery<T> source can be created by calling generation queries provided by ParallelEnumerable, like Range, Repeat, etc., then the other parallel queries can be used subsequently:

internal static void Generation()

IEnumerable<double>sequentialQuery = Enumerable

.Repeat(0, 5) // Output IEnumerable<int>.

.Concat(Enumerable.Range(0, 5)) // Call Enumerable.Concat.

.Where(int32 => int32 > 0) // Call Enumerable.Where.

.Select(int32 => Math.Sqrt(int32)); // Call Enumerable.Select.

ParallelQuery<double> parallelQuery = ParallelEnumerable

.Repeat(0, 5) // Output ParallelQuery<int>.

.Concat(ParallelEnumerable.Range(0, 5)) // Call ParallelEnumerable.Concat.

.Where(int32 => int32 > 0) // Call ParallelEnumerable.Where.

.Select(int32 => Math.Sqrt(int32)); // Call ParallelEnumerable.Select.

}

A PLINQ query can also be started by calling ParallelEnumerable.AsParallel to convert IEnumerable<T>/IEnumerable to ParallelQuery<T>/ParallelQuery:

public static ParallelQuery AsParallel(this IEnumerable source);

public static ParallelQuery<TSource> AsParallel<TSource>(this IEnumerable<TSource> source);

For example,

internal static void AsParallel(IEnumerable<int> source1, IEnumerable source2)

ParallelQuery<int>parallelQuery1 = source1 // IEnumerable<int>.

.AsParallel(); // Output ParallelQuery<int>.

ParallelQuery<int> parallelQuery2 = source2 // IEnumerable.

.AsParallel() // Output ParallelQuery.

.Cast<int>(); // Call ParallelEnumerable.Cast.

}

AsParallel also has an overload accepting a partitioner. Partitioner is discussed in the next chapter.

To use sequential queries for a ParallelQuery<T> source, just call ParallelEnumerable.AsSequential or ParallelEnumerable.AsEnumerable to convert ParallelQuery<T> to IEnumerable<T>, then the sequential queries can be used subsequently:

public static IEnumerable<TSource> AsSequential<TSource>(

this ParallelQuery<TSource> source);

public static IEnumerable<TSource> AsEnumerable<TSource>(

this ParallelQuery<TSource> source);

ParallelEnumerable.AsEnumerable simply calls AsSequential internally, so they are identical. For example:

internal static partial class QueryMethods

private static readonly Assembly CoreLibrary = typeof(object).Assembly;

internal static void SequentialParallel()

IEnumerable<string> obsoleteTypes = CoreLibrary.GetExportedTypes() // Output IEnumerable<Type>.

.AsParallel() // Output ParallelQuery<Type>.

.Where(type => type.GetCustomAttribute<ObsoleteAttribute>() != null) // Call ParallelEnumerable.Where.

.Select(type => type.FullName) // Call ParallelEnumerable.Select.

.AsSequential() // Output IEnumerable<Type>.

.OrderBy(name => name); // Call Enumerable.OrderBy.

obsoleteTypes.WriteLines();

}

The above query can be written in query expression syntax:

internal static void QueryExpression()

IEnumerable<string>obsoleteTypes =

from name in

(from type in CoreLibrary.GetExportedTypes().AsParallel()

where type.GetCustomAttribute<ObsoleteAttribute>() != null

select type.FullName).AsSequential()

orderby name

select name;

obsoleteTypes.WriteLines();

}

Parallel query execution

The foreach statement or the EnumerableEx.ForEach query provided by Ix can be used to sequentially pull the results and start LINQ to Objects query execution. Their parallel version is the ParallelEnumerable.ForAll query.

namespace System.Linq

public static class EnumerableEx

public static void ForEach<TSource>(

this IEnumerable<TSource>source, Action<TSource>onNext);

public static class ParallelEnumerable

public static void ForAll<TSource>(

this ParallelQuery<TSource>source, Action<TSource>action);

}

ForAll can simultaneously pull results from ParallelQuery<T> source with multiple threads, and simultaneously call the specified function on those threads:

internal static void ForEachForAll()

Enumerable

.Range(0, Environment.ProcessorCount * 2)

.ForEach(value => value.WriteLine()); // 0 1 2 3 4 5 6 7

ParallelEnumerable

.Range(0, Environment.ProcessorCount * 2)

.ForAll(value => value.WriteLine()); // 2 6 4 0 5 3 7 1

}

Above is the output after executing the code in a quad core CPU, Unlike ForEach, the values pulled and traced by ForAll is unordered. And if this code runs multiple times, the values can be in different order from time to time. This indeterministic order is the consequence of parallel pulling. The order preservation in parallel query execution is discussed in detail later.

Earlier a WriteLines extension method is defined for IEnumerable<T> as a shortcut to call EnumerableEx.ForEach to pull all values and trace them. The following WriteLines overload can be defined for ParallelQuery<T> to call ParallelEnumerable.ForAll to simply execute parallel query without calling a function for each query result:

public static void WriteLines<TSource>(

this ParallelQuery<TSource> source, Func<TSource, string> messageFactory = null)

if (messageFactory == null)

source.ForAll(value => Trace.WriteLine(value));

else

source.ForAll(value => Trace.WriteLine(messageFactory(value)));

}

Visualizing parallel query execution

It would be nice if the internal execution of sequential/parallel LINQ queries can be visualized with charts. On Windows, Microsoft provides a tool called Concurrency Visualizer for this purpose. It consists of a library of APIs to trace the execution information, and a Visual Studio extension to render the execution information to chart. This tool is very easy and intuitive. Unfortunately, it only renders chart on Windows along with Visual Studio.

Using Concurrency Visualizer

To install the Visual Studio extension, just launch Visual Studio, go to Tools => extensions and Updates… => Online, search “Concurrency Visualizer”, and install. Then restart Visual Studio to complete the installation, and go to Analyze => Concurrency Visualizer => Advanced Settings. In the Filter tab, check Sample Events only:

Then go to Markers tab, check ConcurrencyVisualizer.Markers only:

In Files tab, specified a proper directory for trace files. Notice the trace files can be very large, which depends on how much information is collected.

Next, a reference to Concurrency Visualizer library need to be added to project. Microsoft provides this library as a binary on its web page. For convenience, I have created a NuGet package ConcurrencyVisualizer for .NET Framework and .NET Standard. The library provides the following APIs to render timespans on the time line:

namespace Microsoft.ConcurrencyVisualizer.Instrumentation

public static class Markers

public static Span EnterSpan(int category, string text);

public static MarkerSeries CreateMarkerSeries(string markSeriesName);

public class MarkerSeries

public static Span EnterSpan(int category, string text);

}

The category parameter is used to determine the color of the rendered timespan, and the text parameter is the label for the rendered timespan.

For Linux and macOS, where Visual Studio is not available, the above Marker, MarkerSeries, and Span types can be manually defined to trace text information:

public class Markers

public static Span EnterSpan(int category, string spanName) =>

new Span(category, spanName);

public static MarkerSeries CreateMarkerSeries(string markSeriesName) =>

new MarkerSeries(markSeriesName);

public class Span : IDisposable

private readonly int category;

private readonly string spanName;

private readonly DateTime start;

public Span(int category, string spanName, string markSeriesName = null)

this.category = category;

this.spanName = string.IsNullOrEmpty(markSeriesName)

? spanName : $"{markSeriesName}/{spanName}";

this.start = DateTime.Now;

$"{this.start.ToString("o")}: thread id: {Thread.CurrentThread.ManagedThreadId}, category: {this.category}, span: {this.spanName}".WriteLine();

public void Dispose()

DateTime end = DateTime.Now;

$"{end.ToString("o")}: thread id: {Thread.CurrentThread.ManagedThreadId}, category: {this.category}, span: {this.spanName}, duration: {end – this.start}".WriteLine();

public class MarkerSeries

private readonly string markSeriesName;

public MarkerSeries(string markSeriesName) => this.markSeriesName = markSeriesName;

public Span EnterSpan(int category, string spanName) =>

new Span(category, spanName, this.markSeriesName);

}

If a lot of information is traced, more trace listeners can be optionally added to save the information to file or print to console:

public partial class Markers

static Markers()

// Trace to file:

Trace.Listeners.Add(new TextWriterTraceListener(@"D:\Temp\Trace.txt"));

// Trace to console:

Trace.Listeners.Add(new TextWriterTraceListener(Console.Out));

}

Visualizing sequential and parallel query execution

Now, the Marker, MarkerSeries, and Span types can be used with LINQ queries and ForEach/ForAll to visualize the sequence/parallel execution on Windows, or trace the execution on Linux and macOS:

internal static void RenderForEachForAllSpans()

const string SequentialSpan = nameof(EnumerableEx.ForEach);

// Render a timespan for the entire sequential LINQ query execution, with text label "ForEach".

using (Markers.EnterSpan(-1, SequentialSpan))

MarkerSeries markerSeries = Markers.CreateMarkerSeries(SequentialSpan);

Enumerable.Range(0, Environment.ProcessorCount * 2).ForEach(value =>

// Render a sub timespan for each iteratee execution, with each value as text label.

using (markerSeries.EnterSpan(Thread.CurrentThread.ManagedThreadId, value.ToString()))

// Add workload to extend the iteratee execution to a more visible timespan.

for (int i = 0; i < 10_000_000; i++) { }

value.WriteLine(); // 0 1 2 3 4 5 6 7

});

const string ParallelSpan = nameof(ParallelEnumerable.ForAll);

// Render a timespan for the entire parallel LINQ query execution, with text label "ForAll".

using (Markers.EnterSpan(-2, ParallelSpan))

MarkerSeries markerSeries = Markers.CreateMarkerSeries(ParallelSpan);

ParallelEnumerable.Range(0, Environment.ProcessorCount * 2).ForAll(value =>

// Render a sub timespan for each iteratee execution, with each value as text label.

using (markerSeries.EnterSpan(Thread.CurrentThread.ManagedThreadId, value.ToString()))

// Add workload to extends the iteratee execution to a more visible timespan.

for (int i = 0; i < 10_000_000; i++) { }

value.WriteLine(); // 2 6 4 0 5 3 7 1

});

}

In ForEach and ForAll’s iteratee functions, a for loop of 10 million iterations is executed to add some CPU computing workload to make the function call take longer time, otherwise the rendered timespan of function call can be too small to read. On Windows, click Visual Studio => Analyze => Concurrency Visualizer => Start with Current Project. When the code finishes running, a rich UI is generated. The first tab Utilization shows that the CPU usage was about 25% for a while, which is the sequential LINQ query executing on the quad core CPU. Then the CPU usage became almost 100%, which is the PLINQ execution.

The second tab Threads has the chart of timespans. In the thread list on the left, right click the threads not working on LINQ queries and hide them, so that the chart only has the rendered timespans:

It uncovers how the LINQ queries execute on this quad core CPU. ForEach query pulls the values and call the specified function sequentially with the main thread. ForAll query does the work with 4 threads (main threads and 3 other worker threads), each thread processed 2 values. The values 6, 0, 4, 2 are processed before 7, 1, 5, 3, which leads to the trace output: 2 6 4 0 5 3 7 1.

Click the ForEach timespan, the Current panel shows the execution duration is 4750 milliseconds. Click ForAll, it shows 1314 milliseconds:

This is about 27% of ForEach execution time, which is close to a quarter as expected. It cannot be exactly 25%, because on the device, there are other running processes and threads using CPU, the parallel query also has extra work to manage multithreading, which is covered later in this chapter.

In the last tab Cores, select the LINQ query threads (main thread and other 3 worker thread), and 6760. It shows how the workload is distributed in the 4 cores:

Above LINQ visualization code looks noisy, because it mixes the LINQ query code and the visualization code. Following the Single Responsibility Principle, the visualization can be encapsulated for IEnumerable<T> and ParallelQuery<T>:

internal const string ParallelSpan = "Parallel";

internal const string SequentialSpan = "Sequential";

internal static void Visualize<TSource>(

this IEnumerable<TSource> source,

Action<IEnumerable<TSource>, Action<TSource>> query,

Action<TSource> iteratee, string span = SequentialSpan, int category = -1)

using (Markers.EnterSpan(category, span))

MarkerSeries markerSeries = Markers.CreateMarkerSeries(span);

query(

source,

value =>

using (markerSeries.EnterSpan(Thread.CurrentThread.ManagedThreadId, value.ToString()))

iteratee(value);

});

internal static void Visualize<TSource>(

this ParallelQuery<TSource> source,

Action<ParallelQuery<TSource>, Action<TSource>> query,

Action<TSource> iteratee, string span = ParallelSpan, int category = -2)

using (Markers.EnterSpan(category, span))

MarkerSeries markerSeries = Markers.CreateMarkerSeries(span);

query(

source,

value =>

using (markerSeries.EnterSpan(Thread.CurrentThread.ManagedThreadId, value.ToString()))

iteratee(value);

});

}

And the additional CPU computing workload can also be defined as a function:

internal static int ComputingWorkload(int value = 0, int baseIteration = 10_000_000)

for (int i = 0; i < baseIteration * (value + 1); i++) { }

return value;

}

When it is called as ComputingWorkload() or ComputingWorkload(0), it runs 10 million iterations and output 0; when it is called as ComputingWorkload(1), it runs 20 million iterations and output 1; and so on.

Now the LINQ queries can be visualized in a much cleaner way:

internal static void VisualizeForEachForAll()

Enumerable

.Range(0, Environment.ProcessorCount * 2)

.Visualize(

EnumerableEx.ForEach,

value => (value + ComputingWorkload()).WriteLine());

ParallelEnumerable

.Range(0, Environment.ProcessorCount * 2)

.Visualize(

ParallelEnumerable.ForAll,

value => (value + ComputingWorkload()).WriteLine());

}

Visualizing chaining queries

Besides visualizing query execution with ForEach and ForAll, the following Visualize overloads can be defined to visualize sequential and parallel queries and render their iteratee function execution as timespans, like Select’s selector, Where’s predicate, etc.:

internal static TResult Visualize<TSource, TMiddle, TResult>(

this IEnumerable<TSource> source,

Func<IEnumerable<TSource>, Func<TSource, TMiddle>, TResult> query,

Func<TSource, TMiddle> iteratee,

Func<TSource, string> spanFactory = null,

string span = SequentialSpan)

spanFactory = spanFactory ?? (value => value.ToString());

MarkerSeries markerSeries = Markers.CreateMarkerSeries(span);

return query(

source,

value =>

using (markerSeries.EnterSpan(

Thread.CurrentThread.ManagedThreadId, spanFactory(value)))

return iteratee(value);

});

internal static TResult Visualize<TSource, TMiddle, TResult>(

this ParallelQuery<TSource> source,

Func<ParallelQuery<TSource>, Func<TSource, TMiddle>, TResult> query,

Func<TSource, TMiddle> iteratee,

Func<TSource, string> spanFactory = null,

string span = ParallelSpan)

spanFactory = spanFactory ?? (value => value.ToString());

MarkerSeries markerSeries = Markers.CreateMarkerSeries(span);

return query(

source,

value =>

using (markerSeries.EnterSpan(

Thread.CurrentThread.ManagedThreadId, spanFactory(value)))

return iteratee(value);

});

}

Take a simple Where and Select query chaining as example,

internal static void VisualizeWhereSelect()

Enumerable

.Range(0, 2)

.Visualize(

Enumerable.Where,

value => ComputingWorkload() >= 0, // Where's predicate.

value => $"{nameof(Enumerable.Where)} {value}")

.Visualize(

Enumerable.Select,

value => ComputingWorkload(), // Select's selector.

value => $"{nameof(Enumerable.Select)} {value}")

.WriteLines();

ParallelEnumerable

.Range(0, Environment.ProcessorCount * 2)

.Visualize(

ParallelEnumerable.Where,

value => ComputingWorkload() >= 0, // Where's predicate.

value => $"{nameof(ParallelEnumerable.Where)} {value}")

.Visualize(

ParallelEnumerable.Select,

value => ComputingWorkload(), // Select's selector.

value => $"{nameof(ParallelEnumerable.Select)} {value}")

.WriteLines();

}

The sequential and parallel queries are visualized as:

LINQ to XML in Depth (3) Manipulating XML

Tue, 27 Aug 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to XML in Depth series]

Besides creating and querying XML, LINQ to XML also provides APIs for other XML manipulations, including cloning, deleting, replacing, and updating XML structures:

· Clone

o Explicit Clone: constructors of XAttribute, XCData, XComment, XDeclaration, XDocument, XElement, XProcessingInstruction, XText

· Add

o Add annotations: XObject.AddAnnotation

o Add children: XContainer.Add, XContainer.AddFirst, XStreamingElement.Add

o Add siblings: XNode.AddAfterSelf, XNode.AddBeforeSelf

· Delete

o Delete annotations: XObject.RemoveAnnotations

o Delete attributes: XElement.RemoveAttributes, XAttribute.Remove

o Delete self: XNode.Remove

o Delete children: XContainer.RemoveNodes, XElement.RemoveAll

· Replace

o Replace attributes: XElement.ReplaceAttributes

o Replace self: XNode.ReplaceWith

o Replace children: XContainer.ReplaceNodes, XElement.ReplaceAll

· Update

o Update attribute: XAttribute.Value

o Update comment: XComment.Value

o Update declaration: XDeclaration.Encoding, XDeclaration.Standalone, XDeclaration.Version

o Update document: XDocument.XDeclaration, XDocumentType.InternalSubset, XDocumentType.Name, XDocumentType.PublicId, XDocumentType.SystemId

o Update element: XElement.Name, XElement.Value, XElement.SetAttributeValue, XElement.SetElementValue, XElement.SetValue

.NET Framework also provides APIs for validating and transforming XML:

· Validate with XSD

o Query schema: XAttribute.GetSchemaInfo*, XElement.GetSchemaInfo*

o Validate schema: XAttribute.Validate*, XDocument.Validate*, XElement.Validate*

· Transform with XSL: XslCompiledTransform.Transform

The APIs with * are extension methods provided by System.Xml.Schema.Extensions.

Clone

Most structures can be cloned by calling their constructors with the source instance:

internal static void ExplicitClone()

XElement sourceElement = XElement.Parse("<element />");

XElement clonedElement = new XElement(sourceElement);

XText sourceText = new XText("text");

XText clonedText = new XText(sourceText);

XDocument sourceDocument = XDocument.Load("https://weblogs.asp.net/dixin/rss");

XDocument clonedDocument = new XDocument(sourceDocument);

object.ReferenceEquals(sourceDocument, clonedDocument).WriteLine(); // False

object.Equals(sourceDocument, clonedDocument).WriteLine(); // False

EqualityComparer<XDocument>.Default.Equals(sourceDocument, clonedDocument).WriteLine(); // False

sourceDocument.Equals(clonedDocument).WriteLine(); // False

(sourceDocument == clonedDocument).WriteLine(); // False

XNode.DeepEquals(sourceDocument, clonedDocument).WriteLine(); // True

XNode.EqualityComparer.Equals(sourceDocument, clonedDocument).WriteLine(); // True

}

If an XObject instance is in an XML tree, when it is added to a different XML tree, it is cloned, and the new instance is actually added to the target. The exceptions are XName and XNamespace, which are cached at runtime. For example:

internal static void ImplicitClone()

XElement child = XElement.Parse("<child />");

XName parentName = "parent";

XElement parent1 = new XElement(parentName, child); // Attach.

object.ReferenceEquals(child, parent1.Elements().Single()).WriteLine(); // True

object.ReferenceEquals(parentName, parent1.Name).WriteLine(); // True

XElement parent2 = new XElement(parentName, child); // Clone and attach.

object.ReferenceEquals(child, parent2.Elements().Single()).WriteLine(); // False

object.ReferenceEquals(parentName, parent2.Name).WriteLine(); // True

XElement element = new XElement("element");

element.Add(element); // Clone and attach.

object.ReferenceEquals(element, element.Elements().Single()).WriteLine(); // False

}

Adding, deleting, replacing, updating, and events

Most of APIs to add/replace/delete/update XML structures are very intuitive. And when changing a XObject instance, XObject.Changing and XObject.Changed events are fired before and after the change. For example:

internal static void Manipulate()

XElement child = new XElement("child");

child.Changing += (sender, e) =>

$"Before {e.ObjectChange}: ({sender.GetType().Name} {sender}) => {child}".WriteLine();

child.Changed += (sender, e) =>

$"After {e.ObjectChange}: ({sender.GetType().Name} {sender}) => {child}".WriteLine();

XElement parent = new XElement("parent");

parent.Changing += (sender, e) =>

$"Before {e.ObjectChange}: ({sender.GetType().Name} {sender}) => {parent.ToString(SaveOptions.DisableFormatting)}".WriteLine();

parent.Changed += (sender, e) =>

$"After {e.ObjectChange}: ({sender.GetType().Name} {sender}) => {parent.ToString(SaveOptions.DisableFormatting)}".WriteLine();

child.Value = "value1";

// Before Add: (XText value1) => <child />

// After Add: (XText value1) =>< child>value1</child>

child.Value = "value2";

// Before Remove: (XText value1) =>< child>value1</child>

// After Remove: (XText value1) =>< child />

// Before Add: (XText value2) =>< child />

// After Add: (XText value2) =>< child>value2</child>

child.Value = string.Empty;

// Before Remove: (XText value2) =>< child>value2</child>

// After Remove: (XText value2) =>< child />

// Before Value: (XElement <child />) => <child />

// After Value: (XElement< child></child>) => <child></child>

parent.Add(child);

// Before Add: (XElement <child></child>) =>< parent />

// After Add: (XElement< child></child>) => < parent><child></child></parent>

child.Add(new XAttribute("attribute", "value"));

// Before Add: (XAttribute attribute="value") => <child></child>

// Before Add: (XAttribute attribute="value") => < parent><child></child></parent>

// After Add: (XAttribute attribute="value") => <child attribute="value"></child>

// After Add: (XAttribute attribute="value") => <parent><child attribute="value"></child></parent>

child.AddBeforeSelf(0);

// Before Add: (XText 0) => <parent><child attribute="value"></child></parent>

// After Add: (XText 0) =>< parent>0<child attribute="value"></child></parent>

parent.ReplaceAll(new XText("Text."));

// Before Remove: (XText 0) => <parent>0<child attribute="value"></child></parent>

// After Remove: (XText 0) =>< parent><child attribute="value"></child></parent>

// Before Remove: (XElement <child attribute="value"></child>) => <parent><child attribute="value"></child></parent>

// After Remove: (XElement <child attribute="value"></child>) => <parent />

// Before Add: (XText Text.) =>< parent />

// After Add: (XText Text.) =>< parent>Text.</parent>

parent.Name = "name";

// Before Name: (XElement< parent>Text.</parent>) => <parent>Text.</parent>

// After Name: (XElement< name>Text.</name>) => <name>Text.</name>

XElement clonedChild = new XElement(child);

clonedChild.SetValue(DateTime.Now); // No tracing.

}

There are many APIs to manipulate XML, but there are only 4 kinds of Changing/Changed events: add object, deleting object, update object value, update element/attribute name. For example, as shown above, the APIs to replace objects are shortcuts of deleting old objects and adding new objects. When setting a string as an element’s value, the element first removes its children if there is any, then add the string as a child text node, if the string is not empty string. Also, an object’s events propagate/bubble up to the ancestors, and children and siblings are not impacted. When an object is cloned, the new object’s events is not observed by the original event handlers.

XElement.SetAttributeValue and XElement.SetElementValue are different from other APIs. They can

· add a new attribute/child element if it does not exist

· update the attribute/child element value if it exists:

· remove the attribute/child element if it exists and the provided value to null.

internal static void SetAttributeValue()

XElement element = new XElement("element");

element.Changing += (sender, e) =>

$"Before {e.ObjectChange}: ({sender.GetType().Name} {sender}) => {element}".WriteLine();

element.Changed += (sender, e) =>

$"After {e.ObjectChange}: ({sender.GetType().Name} {sender}) => {element}".WriteLine();

element.SetAttributeValue("attribute", "value1"); // Equivalent to: child1.Add(new XAttribute("attribute", "value1"));

// Before Add: (XAttribute attribute="value1") => <element />

// After Add: (XAttribute attribute="value1") => <element attribute="value1" />

element.SetAttributeValue("attribute", "value2"); // Equivalent to: child1.Attribute("attribute").Value = "value2";

// Before Value: (XAttribute attribute="value1") => <element attribute="value1" />

// After Value: (XAttribute attribute="value2") => <element attribute="value2" />

element.SetAttributeValue("attribute", null);

// Before Remove: (XAttribute attribute="value2") => <element attribute="value2" />

// After Remove: (XAttribute attribute="value2") => <element />

internal static void SetElementValue()

XElement parent = new XElement("parent");

parent.Changing += (sender, e) =>

$"Before {e.ObjectChange}: {sender} => {parent.ToString(SaveOptions.DisableFormatting)}".WriteLine();

parent.Changed += (sender, e) =>

$"After {e.ObjectChange}: {sender} => {parent.ToString(SaveOptions.DisableFormatting)}".WriteLine();

parent.SetElementValue("child", string.Empty); // Add child element.

// Before Add: <child></child> => <parent />

// After Add: <child></child> => <parent><child></child></parent>

parent.SetElementValue("child", "value"); // Update child element.

// Before Value:< child></child> => <parent><child></child></parent>

// After Value: <child /> =>< parent><child /></parent>

// Before Add: value =>< parent><child /></parent>

// After Add: value => < parent><child>value</child></parent>

parent.SetElementValue("child", null); // Remove child element.

// Before Remove:< child>value</child> => < parent><child>value</child></parent>

// After Remove: <child>value</child> => <parent />

}

Annotation

Annotation is not a part of the XML. It is a separate arbitrary data in the memory, and associated with a XObject instance in the memory. The annotation APIs provided by XObject allows adding/querying/deleting any .NET data. Apparently, when cloning or serializing XObject, annotation is ignored on the new XObject and the generated string.

internal static void Annotation()

XElement element = new XElement("element");

element.AddAnnotation(new Uri("https://microsoft.com"));

Uri annotation = element.Annotation<Uri>();

annotation.WriteLine(); // https://microsoft.com

element.WriteLine(); // <element />

XElement clone = new XElement(element); // element is cloned.

clone.Annotations<Uri>().Any().WriteLine(); // False

element.RemoveAnnotations<Uri>();

(element.Annotation<Uri>() == null).WriteLine(); // True

}

Validating XML with XSD

XSD (XML Schema Definition) is the metadata of XML tree, including XML's elements, attributes, constrains rules, etc. System.Xml.Schema.Extensions provides a few APIs to validate XML with provided schema. To obtain a schema, one option is to infer it from existing XML:

public static XmlSchemaSet InferSchema(this XNode source)

XmlSchemaInference schemaInference = new XmlSchemaInference();

using (XmlReader reader = source.CreateReader())

return schemaInference.InferSchema(reader);

}

The returned XmlSchemaSet instance contains s sequence of XmlSchema instances, one for each namespace in the source XML. XmlSchema can be converted to XDocument with the help of XmlWriter:

public static XDocument ToXDocument(this XmlSchema source)

XDocument document = new XDocument();

using (XmlWriter writer = document.CreateWriter())

source.Write(writer);

return document;

}

Still take an RSS feed as example, the following code outputs the RSS feed’s schema:

internal static void InferSchemas()

XDocument aspNetRss = XDocument.Load("https://www.flickr.com/services/feeds/photos_public.gne?id=64715861@N07&format=rss2");

XmlSchemaSet schemaSet = aspNetRss.InferSchema();

schemaSet.Schemas().Cast<XmlSchema>().WriteLines(schema => schema.ToXDocument().ToString());

}

The printed schema is:

<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">

<xs:element name="rss">

<xs:complexType>

<xs:sequence>

<xs:element name="channel">

<xs:complexType>

<xs:sequence>

<xs:element name="title" type="xs:string" />

<xs:element name="link" type="xs:string" />

<xs:element name="description" type="xs:string" />

<xs:element maxOccurs="unbounded" name="item">

<xs:complexType>

<xs:sequence>

<xs:element name="title" type="xs:string" />

<xs:element name="link" type="xs:string" />

<xs:element name="description" type="xs:string" />

<xs:element name="pubDate" type="xs:string" />

<xs:element name="guid">

<xs:complexType>

<xs:simpleContent>

<xs:extension base="xs:string">

<xs:attribute name="isPermaLink" type="xs:boolean" use="required" />

</xs:extension>

</xs:simpleContent>

</xs:complexType>

</xs:element>

<xs:element maxOccurs="unbounded" name="category" type="xs:string" />

</xs:sequence>

</xs:complexType>

</xs:element>

</xs:sequence>

</xs:complexType>

</xs:element>

</xs:sequence>

<xs:attribute name="version" type="xs:decimal" use="required" />

</xs:complexType>

</xs:element>

</xs:schema>

The data is all gone, and there is only structural description for that RSS feed. Save it to a .xsd file, then it can be visualized in Visual Studio’s XML Schema Explorer:

Now, this RSS feed’s schema, represented by XmlSchemaSet, can be used to validate XML. The following example calls the Validate extension methods for XDocument to validate another RSS feed from Flickr. As demonstrated before, Flickr RSS has more elements. Apparently the validation fails:

internal static void Validate()

XDocument aspNetRss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

XmlSchemaSet schemaSet = aspNetRss.InferSchema();

XDocument flickrRss = XDocument.Load("https://www.flickr.com/services/feeds/photos_public.gne?id=64715861@N07&format=rss2");

flickrRss.Validate(

schemaSet,

(sender, args) =>

$"{args.Severity}: ({sender.GetType().Name}) => {args.Message}".WriteLine();

// Error: (XElement) => The element 'channel' has invalid child element 'pubDate'. List of possible elements expected: 'item'.

args.Exception?.WriteLine();

// XmlSchemaValidationException: The element 'channel' has invalid child element 'pubDate'. List of possible elements expected: 'item'.

});

}

Validate has another overload accepting a bool parameter addSchemaInfo. When it is called with true for addSchemaInfo, if an element or attribute is validated, the validation details are saved in an IXmlSchemaInfo instance, and associated with this element or attribute as an annotation. Then, the GetSchemaInfo method can be called on each element or attribute, to query that IXmlSchemaInfo annotation, if available. IXmlSchemaInfo can have a lot of information, including a Validity property, intuitively indicating the validation status:

internal static void GetSchemaInfo()

XDocument aspNetRss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

XmlSchemaSet schemaSet = aspNetRss.InferSchema();

XDocument flickrRss = XDocument.Load("https://www.flickr.com/services/feeds/photos_public.gne?id=64715861@N07&format=rss2");

flickrRss.Validate(schemaSet, (sender, args) => { }, addSchemaInfo: true);

flickrRss

.Root

.DescendantsAndSelf()

.ForEach(element =>

$"{element.XPath()} - {element.GetSchemaInfo()?.Validity}".WriteLine();

element.Attributes().WriteLines(attribute =>

$"{attribute.XPath()} - {attribute.GetSchemaInfo()?.Validity.ToString() ?? "null"}");

});

// /rss - Invalid

// /rss/@version - Valid

// /rss/@xmlns:media - null

// /rss/@xmlns:dc - null

// /rss/@xmlns:creativeCommons - null

// /rss/@xmlns:flickr - null

// /rss/channel - Invalid

// /rss/channel/title - Valid

// /rss/channel/link - Valid

// /rss/channel/description - Valid

// /rss/channel/pubDate - Invalid

// /rss/channel/lastBuildDate - NotKnown

// ...

}

Transforming XML with XSL

XSL (Extensible Stylesheet Language) can transform a XML tree to another. XSL transformation can be done with the System.Xml.Xsl.XslCompiledTransform type:

public static XDocument XslTransform(this XNode source, XNode xsl)

XDocument result = new XDocument();

using (XmlReader sourceReader = source.CreateReader())

using (XmlReader xslReader = xsl.CreateReader())

using (XmlWriter resultWriter = result.CreateWriter())

XslCompiledTransform transform = new XslCompiledTransform();

transform.Load(xslReader);

transform.Transform(sourceReader, resultWriter);

return result;

}

The following example transforms RSS to HTML, the most recent 5 items in RSS are mapped to HTML hyperlinks in an unordered list:

internal static void XslTransform()

XDocument rss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

XDocument xsl = XDocument.Parse(@"

<xsl:stylesheet version='1.0' xmlns:xsl='http://www.w3.org/1999/XSL/Transform'>

<xsl:template match='/rss/channel'>

<ul>

<xsl:for-each select='item[position() &lt;= 5]'><!--Position is less than or equal to 5.-->

<li>

<a>

<xsl:attribute name='href'><xsl:value-of select='link' /></xsl:attribute>

<xsl:value-of select='title' />

</a>

</li>

</xsl:for-each>

</ul>

</xsl:template>

</xsl:stylesheet>");

XDocument html = rss.XslTransform(xsl);

html.WriteLine();

// <ul>

// <li>

// <a href="https://weblogs.asp.net:443/dixin/c-6-0-exception-filter-and-when-keyword">C# 6.0 Exception Filter and when Keyword</a>

// </li>

// <li>

// <a href="https://weblogs.asp.net:443/dixin/use-fiddler-with-node-js">Use Fiddler with Node.js</a>

// </li>

// <li>

// <a href="https://weblogs.asp.net:443/dixin/diskpart-problem-cannot-select-partition">DiskPart Problem: Cannot Select Partition</a>

// </li>

// <li>

// <a href="https://weblogs.asp.net:443/dixin/configure-git-for-visual-studio-2015">Configure Git for Visual Studio 2015</a>

// </li>

// <li>

// <a href="https://weblogs.asp.net:443/dixin/query-operating-system-processes-in-c">Query Operating System Processes in C#</a>

// </li>

// </ul>

}

The above transformation can also be done with LINQ to Objects/XML query:

internal static void Transform()

XDocument rss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

XDocument html = rss

.Element("rss")

.Element("channel")

.Elements("item")

.Take(5)

.Select(item =>

string link = (string)item.Element("link");

string title = (string)item.Element("title");

return new XElement("li", new XElement("a", new XAttribute("href", link), title));

// Equivalent to: return XElement.Parse($"<li><a href='{link}'>{title}</a></li>");

})

.Aggregate(new XElement("ul"), (ul, li) => { ul.Add(li); return ul; }, ul => new XDocument(ul));

html.WriteLine();

}

Summary

This chapter discusses how to use LINQ to XML APIs to work with XML data. LINQ to XML provides types to model XML following a declarative paradigm. After loading XML into memory as objects, they can be queried by LINQ to Objects. LINQ to XML provides additional queries for XML, including DOM navigation, ordering, comparison, etc. LINQ to XML also provide APIs for XPath, as well as XML manipulation, annotation, validation, and transformation, etc.

Installing Android 9 Pie with Microsoft apps on Nexus 7

Tue, 20 Aug 2019 00:00:00 GMT

Years ago I blogged about installing Android 6 on my old Nexus 7 tablet. Now Android 9 is there. This post shows how to install latest Android 9 with latest Microsoft apps.

Increase the system partition size

In old android devices like Nexus 7, the system partition is very small, about 800M. After installing the Android 9 ROM, the Google app store cannot be installed. Even the smallest Open GApps version (pico) fails to install with a 70 error: insufficient space on system partition. So the first step is to increase the system partition size.

TWRP has a resize partition function. However, it does not work for Nexus 7 tablet, since Nexus 7’s system partition is located in the middle of many partitions. So the way is to abandon the original system partition, delete the last partition, and create a new large system partition at the end of the storage space.

First, follow the instructions to download and unzip the ADB tools. For windows, it can be downloaded from https://dl.google.com/android/repository/platform-tools-latest-windows.zip.

Then download and unzip the parted tool from http://iwf1.com/iwf-repo/parted.rar.

Next, follow the instructions to backup the original partition table:

adb push parted /
adb shell
~ # chmod +x parted
~ # /parted /dev/block/mmcblk0
(parted): unit b
(parted): p

Now delete rename to original system partition to something else, delete the last data partition, and recreate new system partition and new data partition:

(parted): name 22 unused1
(parted): rm 30
(parted): mkpart primary 2415919104B  5570068479B
(parted): mkpart primary 5637144576B 31272713727B
(parted): name 30 system
(parted): name 31 userdata
(parted): quit
~ # mke2fs -b 4096 -T ext4 /dev/block/mmcblk0p30
~ # mke2fs -b 4096 -T ext4 /dev/block/mmcblk0p31
~ # /parted /dev/block/mmcblk0 p
~ # exit
adb reboot recovery

After reboot, it is ready to install Android OS and Google app store.

Install Android 9 and Google app store

First, download the latest Android 9 pie ROM, for example, AOSP r46 ROM: https://androidfilehost.com/?fid=6006931924117931258.

Then download the Open GApps, for example, the stock version can work with the enlarged system partition: https://opengapps.org/.

Now transfer these 2 zip file to the device, and use TWRP to install them both.

After reboot, the device will have Android OS and Google app store.

Fix the Pixel Launcher crash

The Google’s Pixel launcher keeps crashing and many users report the same issue. To fix this, the /data/system/packages.xml file need to be edited.

First, use adb to pull the file from device, and edit it with text editor, for example, Visual Studio Code:

adb pull /data/system/packages.xml
code packages.xml

Find the com.google.android.apps.nexuslauncher segment, add 2 permissions:

<package name="com.google.android.apps.nexuslauncher" codePath="/system/priv-app/NexusLauncherPrebuilt" nativeLibraryPath="/system/priv-app/NexusLauncherPrebuilt/lib" publicFlags="877379141" privateFlags="8" ft="16cabc2c4a0" it="16cabc2c4a0" ut="16cabc2c4a0" version="604" userId="10019" isOrphaned="true">
    <sigs count="1" schemeVersion="1">
        <cert index="19" key="308203c7308202afa003020102020900ed6adca108384104300d06092a864886f70d0101050500307a310b30090603550406130255533113301106035504080c0a43616c69666f726e69613116301406035504070c0d4d6f756e7461696e205669657731143012060355040a0c0b476f6f676c6520496e632e3110300e060355040b0c07416e64726f69643116301406035504030c0d6e657875736c61756e63686572301e170d3136303530323231313933325a170d3433303931383231313933325a307a310b30090603550406130255533113301106035504080c0a43616c69666f726e69613116301406035504070c0d4d6f756e7461696e205669657731143012060355040a0c0b476f6f676c6520496e632e3110300e060355040b0c07416e64726f69643116301406035504030c0d6e657875736c61756e6368657230820122300d06092a864886f70d01010105000382010f003082010a0282010100a87aac2fe945313ae9afcc9815d01fa7ede40eaf5a8f280ab4da9c84346b2ca0b2da8d9532d24cb2ce950f6a627273487b7bd9c5601339823842c511926f7e177dc91d919de2e3f6750f9b03fe04f16f008724bbc90fc5e7f2f85e712d523edb93b7903fb007f11756d895d14355d0fa07c05a5773b650b7e6d50e50359884994224ed38591203898840c6b1df6a3604241e24ef9658729e25b6ada3c772a3bbb3d7d832115a2c1901b33b737a5d2a40da9d68c0ecd5e1f9f11d75437286368eab8cd527fe1465aad1da6a0d436767b8b4b53f83e10eac533f48e125b6824899721cabd80cd9749388bd096824e53fce0bcf2c367c6ab9894fa7b135a0b627510203010001a350304e301d0603551d0e041604145055852c1911bb9bd0ba4db3d6a5dfb9acbfa07b301f0603551d230418301680145055852c1911bb9bd0ba4db3d6a5dfb9acbfa07b300c0603551d13040530030101ff300d06092a864886f70d010105050003820101008e190c51ea11edba56bfddfd11ea6d13e729d8ba2a06bd8a346a5d7b7080694003bfe6e284a688f230657d9955854c4dff3fb9d07655911053468ee322314e13e4e098d46d218098d6a316bc0751f1c5b7a7f415358b008cc81ad8b43f05251eb1be5a375c44f8b9c9da46dcfb47f52a7d02dd49e5fe197e257d4946dfc0993f61fe78b4be47c9c6cba1cab164df8d513f41fb79e85f69df1a500e4f0bea4e2f912720765a0a05270f9565e29b46e42f4de5956875f93b7293e9f45ea397c5821738fc2b8bfbf48eb2794dcb7e7997f4a4952fba63605952bf50e8066048afef7fafa9f13ceb122c10759ffbb99d729d671f7c301482fd530cd48a2a9e878f86" />
    </sigs>
    <perms>
        <item name="com.google.android.apps.nexuslauncher.permission.READ_SETTINGS" granted="true" flags="0" />
        <item name="com.google.android.providers.gsf.permission.READ_GSERVICES" granted="true" flags="0" />
        <item name="android.permission.RECEIVE_BOOT_COMPLETED" granted="true" flags="0" />
        <item name="android.permission.BLUETOOTH" granted="true" flags="0" />
        <item name="android.permission.BLUETOOTH_ADMIN" granted="true" flags="0" />
        <item name="android.permission.CONTROL_REMOTE_APP_TRANSITION_ANIMATIONS" granted="true" flags="0" />
        <item name="android.permission.BIND_APPWIDGET" granted="true" flags="0" />
        <item name="android.permission.PACKAGE_USAGE_STATS" granted="true" flags="0" />
        <item name="android.permission.WRITE_SECURE_SETTINGS" granted="true" flags="0" />
        <item name="android.permission.SET_WALLPAPER" granted="true" flags="0" />
        <item name="android.permission.REQUEST_DELETE_PACKAGES" granted="true" flags="0" />
        <item name="android.permission.SET_WALLPAPER_HINTS" granted="true" flags="0" />
        <item name="com.google.android.apps.nexuslauncher.permission.WRITE_SETTINGS" granted="true" flags="0" />
        <item name="com.google.android.apps.nexuslauncher.permission.QSB" granted="true" flags="0" />
        <item name="android.permission.VIEW_INSTANT_APPS" granted="true" flags="0" />
        <item name="com.google.android.launcher.permission.READ_SETTINGS" granted="true" flags="0" />
        <item name="android.permission.STATUS_BAR" granted="true" flags="0" />
        <item name="android.permission.MANAGE_ACTIVITY_STACKS" granted="true" flags="0" />
    </perms>
    <proper-signing-keyset identifier="6" />
</package>

Now push the edited file back to the device:

adb push packages.xml /data/system/packages.xml

After reboot, the Pixel Launcher can work.

Install Microsoft Launcher and apps

Now launch Google app store, install Microsoft Launcher: https://www.microsoft.com/en-us/launcher.

Once it is installed, it places a folder on the desktop, with shortcuts to install the popular Microsoft apps, like OneNote, Skype, Outlook, Teams, etc. Just tab the icons to install them.

LINQ to XML in Depth (2) Query Methods (Operators)

Fri, 16 Aug 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to XML in Depth series]

As fore mentioned, LINQ to XML is just a specialized LINQ to Objects, so all the LINQ to Objects queries can be used in LINQ to XML queries. LINQ to XML provides many additional functions and queries for XML tree navigation, ordering, XPath querying, etc. The following list shows these functions and their return types:

· Navigation queries

o Query direct parent element

§ XObject.Parent -> XElement

o Query all ancestor elements:

§ XNode.Ancestors -> IEnumerable<XElement>

§ XElement.AncestorsAndSelf -> IEnumerable<XElement>

§ IEnumerable<T>.Ancestors* -> IEnumerable<XElement>, where T : XNode

§ IEnumerable<XElement>.AncestorsAndSelf* -> IEnumerable<XElement>

o Query direct child elements

§ XDocument.Root-> XElement

§ XContainer.Element -> XElement

§ XContainer.Elements ->IEnumerable<XElement>

§ IEnumerable<T>.Elements* -> IEnumerable<XElement>, where T : XContainer

o Query direct child nodes

§ XContainer.FirstNode -> XNode

§ XContainer.LastNode -> XNode

§ XContainer.Nodes -> IEnumerable<XNode>

§ IEnumerable<T>.Nodes* -> IEnumerable<XNode>, where T : XContainer

o Query all descendant elements

§ XContainer.Descendants ->IEnumerable<XElement>

§ XElement.DescendantsAndSelf -> IEnumerable<XElement>

§ IEnumerable<T>.Descendants* -> IEnumerable<XElement>, where T : XContainer

§ IEnumerable<XElement>.DescendantsAndSelf* -> IEnumerable<XElement>

o Query all descendant nodes

§ XContainer.DescendantNodes -> IEnumerable<XNode>

§ XElement.DescendantNodesAndSelf -> IEnumerable<XNode>

§ IEnumerable<T>.DescendantNodes* -> IEnumerable<XNode>, where T : XContainer

§ IEnumerable<XElement>.DescendantNodesAndSelf* -> IEnumerable<XNode>

o Query sibling elements

§ XNode.ElementsAfterSelf -> IEnumerable<XElement>

§ XNode.ElementsBeforeSelf -> IEnumerable<XElement>

o Query sibling nodes

§ XNode.PreviousNode -> XNode

§ XNode.NextNode -> XNode

§ XNode.NodesBeforeSelf ->IEnumerable<XNode>

§ XNode.NodesAfterSelf ->IEnumerable<XNode>

o Query attributes

§ XAttribute.PreviousAttribute –> XAttribute

§ XAttribute.NextAttribute -> XAttribute

§ XElement.FirstAttribute -> XAttribute

§ XElement.LastAttribute -> XAttribute

§ XElement.Attribute -> XAttribute

§ XElement.Attributes ->IEnumerable<XAttribute>

§ IEnumerable<XElement>.Attributes* -> IEnumerable<XAttribute>

o Query document

§ XObject.Document –> XDocument

o Query annotations

§ XObject.Annotation<T> –> T, where T : class

§ XObject.Annotations –>IEnumerable<object>

· Ordering queries

o XNode.CompareDocumentOrder -> int

o XNode.IsAfter -> bool

o XNode.IsBefore -> bool

o XNodeDocumentOrderComparer.Compare -> int

o IEnumerable<T>.InDocumentOrder* -> IEnumerable<T>, where T : XNode

· Comparison queries

o XNode.DocumentOrderComparer –> XNodeDocumentOrderComparer

o XNodeDocumentOrderComparer.Compare –> int

o XNode.EqualityComparer –> XNodeEqualityComparer

o XNodeEqualityComparer.Equals –> bool

· XPath queries

o XNode.CreateNavigator** –> XPathNavigator

o XNode.XPathSelectElement** –> XElement

o XNode.XPathSelectElements** –> IEnumerable<XElement>

o XNode.XPathEvaluate** –> object

The functions with * are extension methods provided in static type System.Xml.Linq.Extensions. The functions with ** are extension methods provided in static type System.Xml.XPath.Extensions. The other unmarked methods are instance methods or properties.

Navigation

LINQ to XML provides rich APIs for navigation. And the queries with IEnumerable<XObject> output are also called axis methods or axes. The following example queries the parent element and ancestor element, where. ancestors are parent, parent’s parent, …, recursively:

internal static void ParentAndAncestors()

XElement element = new XElement("element");

XDocument document = new XDocument(new XElement("grandparent", new XElement("parent", element)));

element.Parent.Name.WriteLine(); // parent

element

.Ancestors()

.Select(ancestor => ancestor.Name)

.WriteLines(); // parent grandparent

element

.AncestorsAndSelf()

.Select(selfOrAncestor => selfOrAncestor.Name)

.WriteLines(); // element parent grandparent

object.ReferenceEquals(element.Ancestors().Last(), element.Document.Root).WriteLine(); // True.

}

Notice AncestorsAndSelf first yields self, then yields ancestors recursively. It could be more intuitive if named as SelfAndAncestors.

The following example queries direct child elements. In RSS feed, each <item> can have 0, 1, or multiple tags. And these tags are <category> elements under each <item> element. The following code queries a given RSS feed to get the items with a permalink, then queries the top 5 tags used by these items:

internal static void ChildElements()

XDocument rss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

IEnumerable<string>categories = rss

.Root // <rss>.

.Element("channel") // Single< channel> under <rss>.

.Elements("item") // All< item>s under single <channel>.

.Where(item => (bool)item

.Element("guid") // Single <guid> under each <item>

.Attribute("isPermaLink")) // isPermaLink attribute of <guid>.

.Elements("category") // All <category>s under all <item>s.

.GroupBy(

keySelector: category => (string)category, // String value of each <category>.

elementSelector: category => category,

resultSelector: (key, group) => new { Name = key, Count = group.Count() },

comparer: StringComparer.OrdinalIgnoreCase)

.OrderByDescending(category => category.Count)

.Take(5)

.Select(category => $"[{category.Name}]:{category.Count}");

string.Join(" ", categories).WriteLine();

// [C#]:9 [LINQ]:6 [.NET]:5 [Functional Programming]:4 [LINQ via C#]:4

}

Similar to ancestors, descendants are children, children’s children, …, recursively:

internal static void ChildrenAndDescendants()

XElement root = XElement.Parse(@"

<root>

<![CDATA[cdata]]>0<!--Comment-->

<element>1</element>

<element>2<element>3</element></element>

</root>");

root.Elements()

.WriteLines(element => element.ToString(SaveOptions.DisableFormatting));

// <element>1</element>

// < element>2<element>3</element></element>

root.Nodes()

.WriteLines(node => $"{node.NodeType}: {node.ToString(SaveOptions.DisableFormatting)}");

// CDATA: <![CDATA[cdata]]>

// Text: 0

// Comment: <!--Comment-->

// Element: < element>1</element>

// Element: <element>2<element>3</element></element>

root.Descendants()

.WriteLines(element => element.ToString(SaveOptions.DisableFormatting));

// <element>1</element>

// < element>2<element>3</element></element>

// <element>3</element>

root.DescendantNodes()

.WriteLines(node => $"{node.NodeType}: {node.ToString(SaveOptions.DisableFormatting)}");

// CDATA: <![CDATA[cdata]]>

// Text: 0

// Comment: <!--Comment-->

// Element: < element>1</element>

// Text: 1

// Element: < element>2<element>3</element></element>

// Text: 2

// Element: < element>3</element>

// Text: 3

}

Regarding all the X* types are reference types, when querying the same XML tree, multiple queries’ results from the same source tree can reference to the same instance:

internal static void ResultReferences()

XDocument rss1 = XDocument.Load("https://weblogs.asp.net/dixin/rss");

XElement[] items1 = rss1.Descendants("item").ToArray();

XElement[] items2 = rss1.Element("rss").Element("channel").Elements("item").ToArray();

object.ReferenceEquals(items1.First(), items2.First()).WriteLine(); // True

items1.SequenceEqual(items2).WriteLine(); // True

XDocument rss2 = XDocument.Load("https://weblogs.asp.net/dixin/rss");

XElement[] items3 = rss2.Root.Descendants("item").ToArray();

object.ReferenceEquals(items1.First(), items3.First()).WriteLine(); // False

items1.SequenceEqual(items3).WriteLine(); // False

}

Again, LINQ to XML is just a specialized LINQ to Objects. For example, the implementation of XNode.Ancestors is equivalent to:

namespace System.Xml.Linq

public abstract class XNode : XObject

public IEnumerable<XElement> Ancestors()

for (XElement parent = this.Parent; parent != null; parent = parent.Parent)

yield return parent;

// Other members.

}

And the implementation of the Extensions.Ancestors query is equivalent to:

namespace System.Xml.Linq

public static partial class Extensions

public static IEnumerable<XElement> Ancestors<T>(this IEnumerable<T> source) where T : XNode =>

source

.Where(node => node != null)

.SelectMany(node => node.Ancestors())

// Other members.

}

Ordering

Besides the LINQ to Objects ordering queries, additional ordering queries are provided by LINQ to XML. The InDocumentOrder query orders nodes by their positions in the XML tree, from top node down. For example, above Ancestors yields parent, parent’s parent, …, recursively. InDocumentOrder can reorder them from top down. As a result, the query result is reversed:

internal static void DocumentOrder()

XElement element1 = new XElement("element");

XElement element2 = new XElement("element");

XDocument document = new XDocument(new XElement("grandparent", new XElement("parent", element1, element2)));

element1.IsBefore(element2).WriteLine(); // True

XNode.DocumentOrderComparer.Compare(element1, element2).WriteLine(); // -1

XElement[] ancestors = element1.Ancestors().ToArray();

XNode.CompareDocumentOrder(ancestors.First(), ancestors.Last()).WriteLine(); // 1

ancestors

.InDocumentOrder()

.Select(ancestor => ancestor.Name)

.WriteLines(); // grandparent parent

element1

.AncestorsAndSelf()

.Reverse()

.SequenceEqual(element1.AncestorsAndSelf().InDocumentOrder())

.WriteLine(); // True

}

Apparently, InDocumentOrder requires the source nodes sequence to be in the same XML tree. This is determined by looking up a common ancestor of the source nodes:

internal static void CommonAncestor()

XElement root = XElement.Parse(@"

<root>

<element value='4' />

<element value='2' />

<element value='3'><element value='1' /></element>

</root>");

XElement[] elements = root

.Descendants("element")

.OrderBy(element => (int)element.Attribute("value")).ToArray();

elements.WriteLines(ancestorOrSelf => ancestorOrSelf.ToString(SaveOptions.DisableFormatting));

// <element value="1" />

// <element value="2" />

// <element value="3"><element value="1" /></element>

// <element value="4" />

new XElement[] { elements.First(), elements.Last() }

.InDocumentOrder()

.WriteLines(ancestorOrSelf => ancestorOrSelf.ToString(SaveOptions.DisableFormatting));

// <element value="4" />

// <element value="1" />

new XElement[] { elements.First(), elements.Last(), new XElement("element") }

.InDocumentOrder()

.WriteLines();

// InvalidOperationException: A common ancestor is missing.

}

Notice in the inline XML string, single quotes are used for attribute values, instead of double quotes. This is for readability of C# code, otherwise "" or \" has to be used. According to the W3C XML spec, single quote is legal.

Comparison

LINQ to Objects provides many queries accepting IComparer<T> or IEqualityComparer<T> parameter. To use those queries with XML, LINQ to XML provides 2 built-in comparers:

· XNodeDocumentOrderComparer, which implements IComparer<XNode>. Its Compare method simply calls XNode.CompareDocumentOrder. Its instance is provided by XNode.DocumentOrderComparer property.

· XNodeEqualityComparer, which implements IEqualityComparer<XNode>. Its Equals method simply calls XNode.DeepEquals. Its instance is provided by XNode.EqualityComparer property.

For example, above InDocumentOrder query simply calls OrderBy with XNodeDocumentOrderComparer. Its implementation is equivalent to:

public static partial class Extensions

public static IEnumerable<T> InDocumentOrder<T>(this IEnumerable<T> source) where T : XNode =>

source.OrderBy(node => node, XNode.DocumentOrderComparer);

}

More useful custom queries

With the knowledge of LINQ to Objects and LINQ to XML APIs, more useful queries can be implemented. For example, the following DescendantObjects queries an XObject source’s all descendant XObject instances:

public static partial class XExtensions

public static IEnumerable<XObject> DescendantObjects(this XObject source) =>

Enumerable

.Empty<XObject>()

.Concat(

source is XElement element

? element.Attributes() // T is covariant in IEnumerable<T>.

: Enumerable.Empty<XObject>())

.Concat(

source is XContainer container

? container

.DescendantNodes()

.SelectMany(descendant => EnumerableEx

.Return(descendant)

.Concat(

descendant is XElement descendantElement

? descendantElement.Attributes() // T is covariant in IEnumerable<T>.

: Enumerable.Empty<XObject>()))

: Enumerable.Empty<XObject>());

}

As fore mentioned, XObject can be either node or attribute. So, in the query, If the source is element, it yields the element’s attributes; if the source is XContainer, it yields each descendant node; If a descendant node is element, it yields the attributes.

The following SelfAndDescendantObjects query is intuitive by name and straightforward to implement:

public static IEnumerable<XObject> SelfAndDescendantObjects(this XObject source) =>

EnumerableEx

.Return(source)

.Concat(source.DescendantObjects());

The following Names query finds a XContainer source for all elements’ and attributes’ names:

public static IEnumerable<XName> Names(this XContainer source) =>

(source is XElement element

? element.DescendantsAndSelf()

: source.Descendants())

.SelectMany(descendantElement => EnumerableEx

.Return(descendantElement.Name)

.Concat(descendantElement

.Attributes()

.Select(attribute => attribute.Name)))

.Distinct();

As fore mentioned, XName instances are cached, so Distinct is called to remove the duplicated references.

Above built-in Attributes query finds an element’s attributes. The following AllAttributes queries an XContainer source’s attributes (if it is an element) and all its descendant elements’ attributes:

public static IEnumerable<XAttribute> AllAttributes(this XContainer source) =>

(source is XElement element

? element.DescendantsAndSelf()

: source.Descendants())

.SelectMany(elementOrDescendant => elementOrDescendant.Attributes());

The following Namespaces query finds all namespaces defined in a XContainer source:

public static IEnumerable<(string, XNamespace)> Namespaces(this XContainer source) =>

source // Namespaces are defined as xmlns:prefix="namespace" attributes.

.AllAttributes()

.Where(attribute => attribute.IsNamespaceDeclaration)

.Select(attribute => (attribute.Name.LocalName, (XNamespace)attribute.Value));

It outputs a sequence of (prefix, namespace) tuples, which is very handy. With its help, the following function can be defined to create XmlNamespaceManager from any XContainer source:

public static XmlNamespaceManager CreateNamespaceManager(this XContainer source)

XmlNamespaceManager namespaceManager = new XmlNamespaceManager(new NameTable());

source

.Namespaces()

.ForEach(@namespace => namespaceManager.AddNamespace(@namespace.Item1, @namespace.Item2.ToString()));

return namespaceManager;

}

This function is used later when working with XPath.

XPath

XPath is a simple query language to select or evaluate objects from an XML tree. It consists of 3 parts:

· axis, e.g.:

o / is to select root node (either a document node, or an element node on the fly)

o /rss/channel/item is to select root node, then select root node’s all <rss> direct child elements, then select each < rss> element’s all <channel> child elements, then select each < channel> element’s all <item> child elements

o /rss/@version is to select root node, then select root node’s all <rss> direct child elements, then select each< rss> element’s version attribute

· node test

o text() is to select all text nodes, comment() is to select all comment nodes, etc.

o /element/text() is to select root node, then select all <element> child elements, then select each <element> element’s all child text nodes.

· predicate:

o [1] means select the first node, etc.

o /rss[1]/text()[2] means to select root node, then select the first <rss> child element, then select that <rss> element’s second child text node.

LINQ to XML also provides a few extension methods to work with XPath. The latest XPath version is 3.0, .NET Standard and LINQ to XML implements XPath 1.0.

The CreateNavigator method creates a XmlXPathNavigator, which can be used for navigation and querying:

internal static void XPathNavigator()

XDocument rss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

XPathNavigator rssNavigator = rss.CreateNavigator();

rssNavigator.NodeType.WriteLine(); // Root

rssNavigator.MoveToFirstChild().WriteLine(); // True

rssNavigator.Name.WriteLine(); // rss

((XPathNodeIterator)rssNavigator

.Evaluate("/rss/channel/item[guid/@isPermaLink='true']/category"))

.Cast<XPathNavigator>()

.Select(categoryNavigator => categoryNavigator.UnderlyingObject)

.Cast<XElement>()

.GroupBy(

category => category.Value, // Current text node's value.

category => category,

(key, group) => new { Name = key, Count = group.Count() },

StringComparer.OrdinalIgnoreCase)

.OrderByDescending(category => category.Count)

.Take(5)

.Select(category => $"[{category.Name}]:{category.Count}")

.WriteLines();

// [C#]:9 [LINQ]:6 [.NET]:5 [Functional Programming]:4 [LINQ via C#]:4

}

It implements the same query as previous RSS tags example.

The XPathSelectElements method is a shortcut of calling CreateNavigator to get an XPathNavigator and then call Evaluate. The above query can be shortened as:

internal static void XPathQuery()

XDocument rss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

rss

.XPathSelectElements("/rss/channel/item[guid/@isPermaLink='true']/category")

.GroupBy(

category => category.Value, // Current text node's value.

category => category,

(key, group) => new { Name = key, Count = group.Count() },

StringComparer.OrdinalIgnoreCase)

.OrderByDescending(category => category.Count)

.Take(5)

.Select(category => $"[{category.Name}]:{category.Count}")

.WriteLines();

// [C#]:9 [LINQ]:6 [.NET]:5 [Functional Programming]:4 [LINQ via C#]:4

}

And XPathSelectElement is simply a shortcut of calling XPathSelectElements to get a sequence, then call FirstOrDefault.

XPathEvaluate also calls CreateNavigator and then Evaluate, but it is more flexible. When the XPath is evaluated to a single value, it just returns that value. The following example queries the RSS feed for the average tags count of each <item> element, and also the equivalent LINQ query:

internal static void XPathEvaluateValue()

XDocument rss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

double average1 = (double)rss.XPathEvaluate("count(/rss/channel/item/category) div count(/rss/channel/item)");

average1.WriteLine(); // 4.65

double average2 = rss

.Element("rss")

.Element("channel")

.Elements("item")

.Average(item => item.Elements("category").Count());

average2.WriteLine(); // 4.65

}

When the XPath is evaluated to a sequence of values, XPathEvaluate outputs IEnumerable<object>:

internal static void XPathEvaluateSequence()

XDocument rss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

((IEnumerable<object>)rss

.XPathEvaluate("/rss/channel/item[guid/@isPermaLink='true']/category/text()"))

.Cast<XText>()

.GroupBy(

categoryTextNode => categoryTextNode.Value, // Current text node's value.

categoryTextNode => categoryTextNode,

(key, group) => new { Name = key, Count = group.Count() },

StringComparer.OrdinalIgnoreCase)

.OrderByDescending(category => category.Count)

.Take(5)

.Select(category => $"[{category.Name}]:{category.Count}")

.WriteLines();

// [C#]:9 [LINQ]:6 [.NET]:5 [Functional Programming]:4 [LINQ via C#]:4

}

LINQ to XML also provides overloads for these XPath methods to accept an IXmlNamespaceResolver parameter. When the XPath expression involves namespace, an IXmlNamespaceResolver instance must be provided. Taking another RSS feed from Flickr as an example:

<?xml version="1.0" encoding="utf-8"?>

<rss version="2.0" xmlns:media="http://search.yahoo.com/mrss/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:flickr="urn:flickr:user">

<channel>

<item>

<title>Microsoft Way, Microsoft Campus</title>

<dc:date.Taken>2011-11-02T16:45:54-08:00</dc:date.Taken>

<author flickr:profile="https://www.flickr.com/people/dixin/">nobody@flickr.com (Dixin Yan)</author>

<media:content url="https://farm3.staticflickr.com/2875/9215169916_f8fa57c3da_b.jpg" type="image/jpeg" height="681" width="1024"/>

<media:title>Microsoft Way, Microsoft Campus</media:title>

<media:description type="html">

<p>Microsoft Campus is the informal name of Microsoft's corporate headquarters, located at One Microsoft Way in Redmond, Washington. Microsoft initially moved onto the grounds of the campus on February 26, 1986. <a href="http://en.wikipedia.org/wiki/Microsoft_Redmond_Campus" rel="nofollow">en.wikipedia.org/wiki/Microsoft_Redmond_Campus</a></p>

</media:description>

<media:thumbnail url="https://farm3.staticflickr.com/2875/9215169916_f8fa57c3da_s.jpg" height="75" width="75"/>

<media:credit role="photographer">Dixin Yan</media:credit>

<media:category scheme="urn:flickr:tags">microsoft</media:category>

<!-- Other elements. -->

</item>

<!-- Other items. -->

</channel>

</rss>

It contains additional information than the standard RSS format, and these additional elements/attributes are managed by namespaces. The following example calls the overload of XPathSelectElements to query the media:category elements:

internal static void XPathQueryWithNamespace()

XDocument rss = XDocument.Load("https://www.flickr.com/services/feeds/photos_public.gne?id=64715861@N07&format=rss2");

XmlNamespaceManager namespaceManager = rss.CreateNamespaceManager();

IEnumerable<XElement>query1 = rss.XPathSelectElements("/rss/channel/item/media:category", namespaceManager);

query1.Count().WriteLine(); // 20

IEnumerable<XElement> query2 = rss.XPathSelectElements("/rss/channel/item/media:category");

// XPathException: Namespace Manager or XsltContext needed. This query has a prefix, variable, or user-defined function.

}

Since prefix “media” is in XPath expression, An IXmlNamespaceResolver instance is required. XmlNamespaceManager implements IXmlNamespaceResolver, so simply call the previously defined CreateNamespaceManager method to create it. In contrast, querying the same XPath expression without IXmlNamespaceResolver instance throws XPathException.

The last example calls the overload of XPathEvaluate to query the items’ titles, which has the tag “microsoft” in the< media:category> element:

internal static void XPathEvaluateSequenceWithNamespace()

XDocument rss = XDocument.Load("https://www.flickr.com/services/feeds/photos_public.gne?id=64715861@N07&format=rss2");

((IEnumerable<object>)rss

.XPathEvaluate(

"/rss/channel/item[contains(media:category/text(), 'microsoft')]/media:title/text()",

rss.CreateNamespaceManager()))

.Cast<XText>()

.WriteLines(mediaTitle => mediaTitle.Value);

// Chinese President visits Microsoft

// Satya Nadella, CEO of Microsoft

}

Generate XPath expression

To leverage LINQ to XML, one example is to generate XPath expression for a specified XObject instance, which can be either XAttribute or XNode. The XPath expression can be calculated with the following 3 segments are needed:

the XPath of current object’s parent Element, which can be either calculated recursively, or be provided by caller.
the XPath of current object, which can be

o @attributeName if it is an attribute

o elementName if it is an element

o node test like text(), comment(), etc., if it is any other type of node.

a predicate for current object, which can simply be the position:

o For example, [2] can be used to identify a comment node, if there is another sibling comment node before itself

o also, the position predicate can be omitted if current object has no ambiguous sibling objects, so that XPath of parent object combining XPath of current object selects one single object. For example, if current node is a comment node with no sibling comment node, then parentElement/comment() without position predicate is good enough

First of all, a helper function is needed to calculate the current element or attribute’s name, which should be in simple localName format if the XName instance is not under any namespace, and should be in prefix:localName format if the XName instance is under a namespace. XName.ToString does not work for this requirement, because it returns the {namespaceUri}localName format, as already demonstrated. So, the following XPath extension method can be defined for name:

public static string XPath(this XName source, XElement container)

string prefix = source.Namespace == XNamespace.None

? null

: container.GetPrefixOfNamespace(source.Namespace); // GetPrefixOfNamespace returns null if not found.

return string.IsNullOrEmpty(prefix) ? source.ToString() : $"{prefix}:{source.LocalName}";

}

Regarding the above segment 1 and segment 2 has to be combined, another helper function is needed to combine 2 XPath expressions, which is similar to .NET built-in Combine method provided by System.IO.Path:

private static string CombineXPath(string xPath1, string xPath2, string predicate = null) =>

string.Equals(xPath1, "/", StringComparison.Ordinal) || string.IsNullOrEmpty(xPath2)

? $"{xPath1}{xPath2}{predicate}"

: $"{xPath1}/{xPath2}{predicate}";

Regarding XObject can be either one type of attribute, or several types of nodes, apparently attribute does not need the position predicate, while the different types of nodes all share similar logic to identify the position and the ambiguous siblings. So, the following helper function can be defined for XNode:

private static string XPath<TSource>(

this TSource source,

string parentXPath,

string selfXPath = null,

Func<TSource, bool> siblingPredicate = null) where TSource : XNode

int index = source

.NodesBeforeSelf()

.Cast<TSource>()

.Where(siblingPredicate ?? (_ => true))

.Count();

string predicate = index == 0

&& !source

.NodesAfterSelf()

.Cast<TSource>()

.Where(siblingPredicate ?? (_ => true))

.Any()

? null

: $"[{index + 1}]";

return CombineXPath(parentXPath, selfXPath, predicate);

}

Now, the following XPath extension method can be defined to generate XPath expression for an element:

public static string XPath(this XElement source, string parentXPath = null) =>

string.IsNullOrEmpty(parentXPath) && source.Parent == null && source.Document == null

? "/" // source is an element on the fly, not attached to any parent node.

: source.XPath(

parentXPath ?? source.Parent?.XPath(),

source.Name.XPath(source),

sibling => sibling.Name == source.Name);

There is a special case for element. As fore mentioned, an element can be constructed on the fly, and it is the root node of its XML tree. In this case, just outputs XPath root expression /. For other cases, just call above XPath extension method for XNode, with:

· XPath of parent element, if not provided then calculate recursively

· XPath of element name, which can be generated by calling above XPath extension method for XName

· A lambda expression to identify ambiguous sibling elements with the same element name, so that the proper XPath predicate can be generated

The XPath overloads for comment/text/processing instruction nodes are straightforward:

public static string XPath(this XComment source, string parentXPath = null) =>

source.XPath(parentXPath ?? source.Parent?.XPath(), "comment()");

public static string XPath(this XText source, string parentXPath = null) =>

source.XPath(parentXPath ?? source.Parent?.XPath(), "text()");

public static string XPath(this XProcessingInstruction source, string parentXPath = null) =>

source.XPath(

parentXPath ?? source.Parent?.XPath(),

$"processing-instruction('{source.Target}')",

sibling => string.Equals(sibling.Target, source.Target, StringComparison.Ordinal));

And the XPath overload for attribute just combine parent element’s XPath with the format of @attributeName:

public static string XPath(this XAttribute source, string parentXPath = null) =>

CombineXPath(parentXPath ?? source.Parent?.XPath(), $"@{source.Name.XPath(source.Parent)}");

Here are some examples of using these extension methods:

internal static void GenerateXPath()

XDocument aspNetRss = XDocument.Load("https://weblogs.asp.net/dixin/rss");

XElement element1 = aspNetRss

.Root

.Element("channel")

.Elements("item")

.Last();

element1.XPath().WriteLine(); // /rss/channel/item[20]

XElement element2 = aspNetRss.XPathSelectElement(element1.XPath());

object.ReferenceEquals(element1, element2).WriteLine(); // True

XDocument flickrRss = XDocument.Load("https://www.flickr.com/services/feeds/photos_public.gne?id=64715861@N07&format=rss2");

XAttribute attribute1 = flickrRss

.Root

.Descendants("author") // <author flickr:profile="https://www.flickr.com/people/dixin/">...</author>.

.First()

.Attribute(XName.Get("profile", "urn:flickr:user")); // <rss xmlns:flickr="urn:flickr:user">...</rss>.

attribute1.XPath().WriteLine(); // /rss/channel/item[1]/author/@flickr:profile

XAttribute attribute2 = ((IEnumerable<object>)flickrRss

.XPathEvaluate(attribute1.XPath(), flickrRss.CreateNamespaceManager()))

.Cast<XAttribute>()

.Single();

object.ReferenceEquals(attribute1, attribute2).WriteLine(); // True

}

LINQ to XML in Depth (1) Modeling XML

Sat, 10 Aug 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to XML in Depth series]

XML (eXtensible Markup Language) is widely used to represent, store, and transfer data. .NET Standard provides LINQ to XML APIs to query XML data source. LINQ to XML APIs are located in System.Xml.XDocument NuGet package for .NET Core, and System.Xml.Linq.dll assembly for .NET Framework. LINQ to XML can be viewed as specialized LINQ to Objects, where the queried objects represent XML structures.

Modeling XML

.NET Standard provides a set provides types to represent the XML structures, including XML document, namespace, element, attribute, comment, text, etc., so that XML can be loaded to memory as instances of those types, and then be queried with LINQ as objects.

Imperative paradigm vs. declarative paradigm

The XML DOM APIs are provided since .NET Framework 1.0. There a set of Xml* types in System.Xml namespace representing XML structures. The following list shows their inheritance hierarchy:

· XmlNamedNodeMap

o XmlAttributeCollection

· XmlNode

o XmlAttribute

o XmlDocument

o XmlDocumentFragment

o XmlEntity

o XmlLinkedNode

§ XmlCharacterData

§ XmlCDataSection

§ XmlComment

§ XmlSignificantWhitespace

§ XmlText

§ XmlWhitespace

§ XmlDeclaration

§ XmlDocumentType

§ XmlElement

§ XmlEntityReference

§ XmlProcessingInstruction

o XmlNotation

· XmlNodeList

· XmlQualifiedName

These DOM APIs for XML can be used to model and manipulate XML structures in imperative paradigm. Take the following XML fragment as example:

<channel>

<item>

<title>LINQ via C#</title>

<link>https://weblogs.asp.net/dixin/linq-via-csharp</link>

<description>

<p>This is a tutorial of LINQ and functional programming. Hope it helps.</p>

</description>

<pubDate>Mon, 07 Sep 2009 00:00:00 GMT</pubDate>

<guid isPermaLink="true">https://weblogs.asp.net/dixin/linq-via-csharp</guid>

<category>C#</category>

<category>LINQ</category>

<!--Comment.-->

<dixin:source>https://github.com/Dixin/CodeSnippets/tree/master/Dixin/Linq</dixin:source>

</item>

</channel>

</rss>

It is a small RSS feed sample with one single item. The following example calls XML DOM APIs to build such a XML tree, and serialize the XML tree to string:

internal static void CreateAndSerializeWithDom()

XmlNamespaceManager namespaceManager = new XmlNamespaceManager(new NameTable());

const string NamespacePrefix = "dixin";

namespaceManager.AddNamespace(NamespacePrefix, "https://weblogs.asp.net/dixin");

XmlDocument document = new XmlDocument(namespaceManager.NameTable);

XmlElement rss = document.CreateElement("rss");

rss.SetAttribute("version", "2.0");

XmlAttribute attribute = document.CreateAttribute(

"xmlns", NamespacePrefix, namespaceManager.LookupNamespace("xmlns"));

attribute.Value = namespaceManager.LookupNamespace(NamespacePrefix);

rss.SetAttributeNode(attribute);

document.AppendChild(rss);

XmlElement channel = document.CreateElement("channel");

rss.AppendChild(channel);

XmlElement item = document.CreateElement("item");

channel.AppendChild(item);

XmlElement title = document.CreateElement("title");

title.InnerText = "LINQ via C#";

item.AppendChild(title);

XmlElement link = document.CreateElement("link");

link.InnerText = "https://weblogs.asp.net/dixin/linq-via-csharp";

item.AppendChild(link);

XmlElement description = document.CreateElement("description");

description.InnerXml = "<p>This is a tutorial of LINQ and functional programming. Hope it helps.</p>";

item.AppendChild(description);

XmlElement pubDate = document.CreateElement("pubDate");

pubDate.InnerText = new DateTime(2009, 9, 7).ToString("r");

item.AppendChild(pubDate);

XmlElement guid = document.CreateElement("guid");

guid.InnerText = "https://weblogs.asp.net/dixin/linq-via-csharp";

guid.SetAttribute("isPermaLink", "true");

item.AppendChild(guid);

XmlElement category1 = document.CreateElement("category");

category1.InnerText = "C#";

item.AppendChild(category1);

XmlNode category2 = category1.CloneNode(false);

category2.InnerText = "LINQ";

item.AppendChild(category2);

XmlComment comment = document.CreateComment("Comment.");

item.AppendChild(comment);

XmlElement source = document.CreateElement(NamespacePrefix, "source", namespaceManager.LookupNamespace(NamespacePrefix));

source.InnerText = "https://github.com/Dixin/CodeSnippets/tree/master/Dixin/Linq";

item.AppendChild(source);

// Serialize XmlDocument to string.

// rssItem.ToString() outputs "System.Xml.XmlElement".

// rssItem.OuterXml outputs a single line of XML text.

StringBuilder xmlString = new StringBuilder();

XmlWriterSettings settings = new XmlWriterSettings

Indent = true,

IndentChars = " ",

OmitXmlDeclaration = true

};

using (XmlWriter writer = XmlWriter.Create(xmlString, settings))

document.Save(writer);

xmlString.WriteLine();

}

These APIs have a few disadvantages:

· Any XML structure has to be created with a XmlDocument instance.

· XML tree has to be built imperatively, node by node.

· Additional work is needed to manage namespaces and prefixes.

· Some operations, like serialization, is not straightforward.

Fortunately, LINQ to XML does not work with these Xml* types. It redesigns a bunch of X* types under System.Xml.Linq namespace, and enables LINQ queries for these objects. The following list shows the inheritance hierarchy of all the X* types, as well as each type’s conversion from/to other types, and their overloaded operators:

· XDeclaration

· XName: implicit convertible from string, ==, !=

· XNamespace: implicit convertible from string, + string, ==, !=

· XObject

o XAttribute: explicit convertible to string/bool/bool?/int/int?/uint/uint?/long/long?/ulong/ulong?/float/float?/double/double?/decimal/decimal?/DateTime/DateTime?/TimeSpan/TimeSpan?/Guid/Guid?

o XNode: DeepEquals

§ XComment

§ XContainer

§ XDocument

§ XElement: explicit convertible to string/bool/bool?/int/int?/uint/uint?/long/long?/ulong/ulong?/float/float?/double/double?/decimal/decimal?/DateTime/DateTime?/TimeSpan/TimeSpan?/Guid/Guid?

§ XDocumentType

§ XProcessingInstruction

§ XText

§ XCData

· XStreamingElement

As the names suggest, e.g., XNode represents a XML node, XDocument represents a XML document, XName represents XML element name or XML attribute name, etc. And apparently, An XML element/attribute name is essentially a string, so XName implements implicit conversion from string, which provides great convenience. The following example builds the same XML tree with the new LINQ to XML types:

internal static void CreateAndSerializeWithLinq()

XNamespace @namespace = "https://weblogs.asp.net/dixin";

XElement rss = new XElement(

"rss",

new XAttribute("version", "2.0"),

new XAttribute(XNamespace.Xmlns + "dixin", @namespace),

new XElement(

"channel",

new XElement(

"item", // Implicitly converted to XName.

new XElement("title", "LINQ via C#"),

new XElement("link", "https://weblogs.asp.net/dixin/linq-via-csharp"),

new XElement(

"description",

XElement.Parse("<p>This is a tutorial of LINQ and functional programming. Hope it helps.</p>")),

new XElement("pubDate", new DateTime(2009, 9, 7).ToString("r")),

new XElement(

"guid",

new XAttribute("isPermaLink", "true"), // "isPermaLink" is implicitly converted to XName.

"https://weblogs.asp.net/dixin/linq-via-csharp"),

new XElement("category", "C#"),

new XElement("category", "LINQ"),

new XComment("Comment."),

new XElement(

@namespace + "source",

"https://github.com/Dixin/CodeSnippets/tree/master/Dixin/Linq"))));

rss.ToString().WriteLine(); // Serialize XDocument to string.

}

The new APIs is shorter and more intuitive:

· XML structure can be created on the fly, XDocument is not involved in the entire example.

· XML tree can be built declaratively.

· Easier namespace management, with prefix automatically taken care of.

· To serialize an XML tree, simply call ToString.

Types, conversions and operators

Besides XDocument, XElement, XAttribute, and XComment in above example, some other XML structures can also can declaratively constructed too:

internal static void Construction()

XDeclaration declaration = new XDeclaration("1.0", null, "no");

declaration.WriteLine(); // <?xml version="1.0" standalone="no"?>

XDocumentType documentType = new XDocumentType("html", null, null, null);

documentType.WriteLine(); // <!DOCTYPE html>

XText text = new XText("<p>text</p>");

text.WriteLine(); // & lt;p&gt;text&lt;/p&gt;

XCData cData = new XCData("cdata");

cData.WriteLine(); // <![CDATA[cdata]]>

XProcessingInstruction processingInstruction = new XProcessingInstruction(

"xml-stylesheet", @"type=""text/xsl"" href=""Style.xsl""");

processingInstruction.WriteLine(); //< ?xml-stylesheet type="text/xsl" href="Style.xsl"?>

}

XName is different. LINQ to XML provides 2 equivalent ways to instantiate XName:

· calling XName.Get

· implicitly converting from string (which is implemented with XName.Get as well).

The constructor is not exposed, because LINQ to XML caches all the constructed XName instances at runtime, so a XName instance is constructed only once for a specific name. LINQ to XML also implements the == and != operator by checking the reference equality:

internal static void Name()

XName attributeName1 = "isPermaLink"; // Implicitly convert string to XName.

XName attributeName2 = XName.Get("isPermaLink");

XName attributeName3 = "IsPermaLink";

object.ReferenceEquals(attributeName1, attributeName2).WriteLine(); // True

(attributeName1 == attributeName2).WriteLine(); // True

(attributeName1 != attributeName3).WriteLine(); // True

}

XNamespace has the same behaviour as XName. additionally, it implements the + operator to combine the namespace and local name:

internal static void Namespace()

XNamespace namespace1 = "http://www.w3.org/XML/1998/namespace"; // Implicitly convert string to XNamespace.

XNamespace namespace2 = XNamespace.Xml;

XNamespace namespace3 = XNamespace.Get("http://www.w3.org/2000/xmlns/");

(namespace1 == namespace2).WriteLine(); // True

(namespace1 != namespace3).WriteLine(); // True

XNamespace @namespace = "https://weblogs.asp.net/dixin";

XName name = @namespace + "localName"; // + operator.

name.WriteLine(); // {https://weblogs.asp.net/dixin}localName

XElement element = new XElement(name, new XAttribute(XNamespace.Xmlns + "dixin", @namespace)); // + operator.

element.WriteLine(); // <dixin:localName xmlns:dixin="https://weblogs.asp.net/dixin" />

}

XElement can be explicitly converted to .NET primitive types, e.g.:

internal static void Element()

XElement pubDateElement = XElement.Parse("<pubDate>Mon, 07 Sep 2009 00:00:00 GMT</pubDate>");

DateTime pubDate = (DateTime)pubDateElement;

pubDate.WriteLine(); // 9/7/2009 12:00:00 AM

}

The above conversion is implemented by calling DateTime.Parse with the string value returned by XElement.Value.

XAttribute can be converted to primitive types too:

internal static void Attribute()

XName name = "isPermaLink";

XAttribute isPermanentLinkAttribute = new XAttribute(name, "true");

bool isPermaLink = (bool)isPermanentLinkAttribute;

isPermanentLink.WriteLine() // True

}

Here the conversion is implemented by calling System.Xml.XmlConvert’s ToBoolean method with the string value returned by XElement.Value.

XComment, XDocument, XElement, XDocumentType, XProcessingInstruction, XText, and XCData types inherit XNode. XNode provides a DeepEquals method to compare any 2 nodes:

internal static void DeepEquals()

XElement element1 = XElement.Parse("<parent><child></child></parent>");

XElement element2 = new XElement("parent", new XElement("child")); // < parent><child /></parent>

object.ReferenceEquals(element1, element2).WriteLine(); // False

XNode.DeepEquals(element1, element2).WriteLine(); // True

XElement element3 = new XElement("parent", new XElement("child", string.Empty)); // < parent><child></child></parent>

object.ReferenceEquals(element1, element2).WriteLine(); // False

XNode.DeepEquals(element1, element3).WriteLine(); // False

}

Here element2’s child element is constructed with null content, so it is an empty element node <child /> (where XElement.IsEmpty returns true). element3’s child element is constructed with an empty string as content, so it is a non-empty element< child></child> ((where XElement.IsEmpty returns false). As a result, element1 has the same node structures and node values as element2, and they are different from element3.

Read and deserialize XML

In LINQ to XML, XML can be easily read or deserialized to XNode/XElement/XDocument instances in memory. with the following APIs:

· XmlReader (under System.Xml namespace)

· XNode.CreateReader, XNode.ReadFrom

· XDocument.Load, XDocument.Parse

· XElement.Load, XElement.Parse

The APIs accepting URI, for example:

internal static void Read()

using (XmlReader reader = XmlReader.Create("https://weblogs.asp.net/dixin/rss"))

reader.MoveToContent();

XNode node = XNode.ReadFrom(reader);

XElement element1 = XElement.Parse("<html><head></head><body></body></html>");

XElement element2 = XElement.Load("https://weblogs.asp.net/dixin/rss");

XDocument document1 = XDocument.Parse("<html><head></head><body></body></html>");

XDocument document2 = XDocument.Load("https://microsoft.com"); // Succeed.

XDocument document3 = XDocument.Load("https://asp.net"); // Fail.

// System.Xml.XmlException: The 'ul' start tag on line 68 position 116 does not match the end tag of 'div'. Line 154, position 109.

}

Reading an RSS feed to construct an XML tree usually work smoothly, since RSS is just XML. Reading a web page usually has bigger chance to fail, because in the real world, a HTML document may be not strictly structured.

The above example reads entire XML document and deserialize the string to XML tree in the memory. Regarding the specified XML can have arbitrary size, XmlReader and XNode.ReadFrom can also read XML fragment by fragment:

internal static IEnumerable<XElement> RssItems(string rssUri)

using (XmlReader reader = XmlReader.Create(rssUri))

reader.MoveToContent();

while (reader.Read())

if (reader.NodeType == XmlNodeType.Element && reader.Name.Equals("item", StringComparison.Ordinal))

yield return (XElement)XNode.ReadFrom(reader);

}

As discussed in the LINQ to Objects chapter, function with yield return statement is compiled to generator construction, and all the API calls in above function body is deferred, so each <item> in the RSS feed is read and deserialized on demand.

Serialize and write XML

The following APIs are provided to serialize XML to string, or write XML to somewhere (file system, memory, etc.):

· XmlWriter

· XObject.ToString

· XNode.ToString, XNode.WriteTo

· XContainer.CreateWriter

· XDocument.Save

· XElement.Save

· XStramingElement.Save, XStramingElement.ToString, XStreamingElement.WriteTo

For example:

internal static void Write()

XDocument document1 = XDocument.Load("https://weblogs.asp.net/dixin/rss");

using (FileStream stream = File.OpenWrite(Path.GetTempFileName()))

document1.Save(stream);

XElement element1 = new XElement("element", string.Empty);

XDocument document2 = new XDocument();

using (XmlWriter writer = document2.CreateWriter())

element1.WriteTo(writer);

document2.WriteLine(); //< element></element>

XElement element2 = new XElement("element", string.Empty);

using (XmlWriter writer = element2.CreateWriter())

writer.WriteStartElement("child");

writer.WriteAttributeString("attribute", "value");

writer.WriteString("text");

writer.WriteEndElement();

element2.ToString(SaveOptions.DisableFormatting).WriteLine();

// <element><child attribute="value">text</child></element>

}

XNode also provides a ToString overload to accept a SaveOptions flag:

internal static void XNodeToString()

XDocument document = XDocument.Parse(

"<root xmlns:prefix='namespace'><element xmlns:prefix='namespace' /></root>");

document.ToString(SaveOptions.None).WriteLine(); // Equivalent to document.ToString().

// <root xmlns:prefix="namespace">

// <element xmlns:prefix="namespace" />

// </root>

document.ToString(SaveOptions.DisableFormatting).WriteLine();

// <root xmlns:prefix="namespace"><element xmlns:prefix="namespace" /></root>

document.ToString(SaveOptions.OmitDuplicateNamespaces).WriteLine();

// <root xmlns:prefix="namespace">

// <element />

// </root>

}

To serialize XML with even more custom settings, the XmlWriter with XmlWriterSettings approach in the DOM API example can be used.

Deferred construction

The XStreamingElement is a special type. It is used to defer the build of element. For example:

internal static void StreamingElementWithChildElements()

IEnumerable<XElement> ChildElementsFactory() =>

Enumerable

.Range(0, 5).Do(value => value.WriteLine())

.Select(value => new XElement("child", value));

XElement immediateParent = new XElement("parent", ChildElementsFactory()); // 0 1 2 3 4.

immediateParent.ToString(SaveOptions.DisableFormatting).WriteLine();

// < parent><child>0</child><child>1</child><child>2</child><child>3</child><child>4</child></parent>

XStreamingElement deferredParent = new XStreamingElement("parent", ChildElementsFactory()); // Deferred.

deferredParent.ToString(SaveOptions.DisableFormatting).WriteLine();

// 0 1 2 3 4

// < parent><child>0</child><child>1</child><child>2</child><child>3</child><child>4</child></parent>

}

Here a factory function is defined to generate a sequence of child elements. It calls the Do query from Interactive Extension (Ix) to prints each value when that pulled from the sequence. Next, the XElement constructor is called, which immediately pulls all child elements from the sequence returned by the factory function, so that the parent element is immediately built with those child elements. Therefore, the Do query is executed right away, and prints the values of the generated child elements. In contrast, XStreamingElement constructor does not pull the child elements from the sequence, the values are not printed yet by Do. The pulling is deferred until the parent element needs to be built, for example, when XStreamingElement.Save/XStreamingElement.ToString/XStreamingElement.WriteTo is called.

This feature can also be demonstrated by modifying the child elements. For XElement, once constructed, the element is built immediately, and is not impacted by modifying the original child elements. In contrast, XStreamingElement can be impacted by the modification:

internal static void StreamingElementWithChildElementModification()

XElement source = new XElement("source", new XElement("child", "a"));

XElement child = source.Elements().Single();

XElement immediateParent = new XElement("parent", child);

XStreamingElement deferredParent = new XStreamingElement("parent", child); // Deferred.

child.Value = "b";

immediateParent.ToString(SaveOptions.DisableFormatting).WriteLine(); // < parent><child>a</child></parent>

deferredParent.ToString(SaveOptions.DisableFormatting).WriteLine(); // < parent><child>b</child></parent>

}

Setup Open Live Writer and sync with Windows Live Writer cross computers

Mon, 22 Jul 2019 00:00:00 GMT

Today I am setting up a new PC. I use Windows Live Writer to write for this blog for years, and found the new installation can no longer work for my blog. I tried Open Live Writer. Fortunately Open Live Writer works.

The next step is to synchronize the new Open Live Writer installation with my other computer’s Windows Live Writer.

Plugins

The plugins for Windows Live Writer does not work for Open Live Writer. Take the the VSPaste plugin as example, it can paste code copied from Visual Studio into Windows Live Writer and keep the style and color. It is built for Windows Live Writer and does not work for Open Live Writer.

To rebuild it for Open Live Writer, first decompile the plugin dll file. Then replace the reference assembly WindowsLive.Writer.Api.dll with OpenLiveWriter.Api.dll, which can be found at %UserProfile%\AppData\Local\OpenLiveWriter\app-0.6.2.

The rebuilt plugin VSPaste.OpenLiveWriter.dll should be placed in the %UserProfile%\AppData\Local\OpenLiveWriter\app-0.6.2\Plugins directory.

To debug the rebuilt plugin, clone the source of Open Live Writer from https://github.com/OpenLiveWriter/OpenLiveWriter.git and open in Visual Studio. Then launch Open Live Writer, and use Visual Studio to attach to the Open Live Writer process.

I have uploaded the plugin source to GitHub for demonstration purpose: https://github.com/Dixin/LiveWriter.VSPaste.

To synchronize the plugin cross multiple PCs, I put the plugins (VSPaste.WindowsLiveWriter.dll and VSPaste.OpenLiveWriter.dll) into a OneDrive directory, for example, D:\OneDrive\LiveWriter\Plugins. Then create a junction point for Windows Live Writer and Open Live Writer:

mklink /J C:\Users\dixin\AppData\Local\OpenLiveWriter\app-0.6.2\Plugins D:\OneDrive\LiveWriter\Plugins

Synchronize blog id

As mentioned in an earlier post, the blog account is associated with a GUID, and the GUID is written to blog post file (which is not a good design). For Windows Live Writer, the blog id is located under HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows Live\Writer\Weblog\. For Open Live Writer, it is located under Computer\HKEY_CURRENT_USER\Software\OpenLiveWriter\Weblogs. For the same blog, just make sure the same blog account has the same GUID.

C:\Users\dixinyan\AppData\Roaming\Windows Live Writer\blogtemplates

Synchronize configurations

The configurations for Windows Live Writer is located under %UserProfile%\AppData\Roaming\Windows Live Writer, and the configuratiuons for Open Live Writer is located under %UserProfile%\AppData\Roaming\OpenLiveWriter. The configurations includes blog template, user dictionary for spell check, etc. These can also be synchronized cross multiple PCs using OneDrive:

mklink /J C:\Users\dixin\AppData\Roaming\OpenLiveWriter D:\OneDrive\LiveWriter\Configurations

Synchronize blog posts

Once blog id is synchronized, it is safe to synchronize blog posts cross multiple PCs, again, using OneDrive:

mklink /J “C:\Users\dixin\Documents\My Weblog Posts” “D:\OneDrive\LiveWriter\My Weblog Posts”

LINQ to Objects in Depth (7) Building Custom Query Methods

Sun, 07 Jul 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to Objects in Depth series]

With the understanding of standard queries in .NET Standard and the additional queries provided by Microsoft, it is easy to define custom LINQ queries for objects. This chapter demonstrates how to define the following useful LINQ to Object queries:

· Sequence queries: output a new IEnumerable<T> sequence (deferred execution)

o Generation: Create, Guid, RandomInt32, RandomDouble, FromValue, EmptyIfNull

o Concatenation: ConcatJoin

o Partitioning: Subsequence, Pagination

o Ordering: OrderBy*, OrderByDescending*, ThenBy*, ThenByDescending*

o Grouping, Join, Set: GroupBy*, Join*, GroupJoin*, Distinct, Union, Intersect*, Except*

o List: Insert, Remove, RemoveAll, RemoveAt

· Collection queries: output a new collection (immediate execution)

o Conversion: ToDictionary, ToLookup

· Value queries: output a single value (immediate execution)

o Aggregation: PercentileExclusive, PercentileInclusive, Percentile

o Quantifiers: IsNullOrEmpty, Contains

o Equality: SequenceEqual

o List: IndexOf, LastIndexOf

· Void queries: no output (immediate execution)

o Iteration: ForEach

Just like the standard and Ix queries, all the above sequence queries implement deferred execution, where the sequence queries marked with * implements eager evaluation, and other unmarked sequence queries implements lazy evaluation. All the other collection queries, value queries, and void queries implement immediate execution.

These queries can be defined in the following static class EnumerableX:

public static partial class EnumerableX { }

Sequence queries

Generation

Ix provides a Create query to execute sequence factory function once. In contrast, the following Create overload is defined to generate a sequence of values by repeatedly calling a value factory:

public static IEnumerable<TResult> Create<TResult>(

Func<TResult>valueFactory, int? count = null)

if (count < 0)

throw new ArgumentOutOfRangeException(nameof(count));

IEnumerable<TResult>CreateGenerator()

if (count == null)

while (true)

yield return valueFactory(); // Deferred execution.

for (int index = 0; index < count; index++)

yield return valueFactory(); // Deferred execution.

return CreateGenerator();

}

When count is not provided, an infinite sequence is generated. For example, the following Guid query uses Create to repeatedly call Guid.NewGuid, so that it generates a sequence of new GUIDs:

public static IEnumerable<Guid> NewGuid(int? count) => Create(Guid.NewGuid, count);

The following queries generate a sequence of random numbers:

public static IEnumerable<int> RandomInt32(

int min, int max, int? count = null, int? seed = null) =>

EnumerableEx.Defer(() =>

Random random = new Random(seed ?? Environment.TickCount);

return Create(() => random.Next(min, max), count);

});

public static IEnumerable<double> RandomDouble(int? count = null, int ? seed = null) =>

EnumerableEx.Defer(() =>

Random random = new Random(seed ?? Environment.TickCount);

return Create(random.NextDouble, count);

});

Here Defer is called to defer the instantiation of Random.

The following EmptyIfNull can be used to omit null checks:

public static IEnumerable<TSource> EmptyIfNull<TSource>(this IEnumerable<TSource> source) =>

source ?? Enumerable.Empty<TSource>();

For example:

internal static void EmptyIfNull(IEnumerable<int> source1, IEnumerable<int> source2)

IEnumerable<int>positive = source1.EmptyIfNull()

.Union(source2.EmptyIfNull())

.Where(int32 => int32 > 0);

}

Concatenation

string has a useful method Join:

namespace System

public class String

public static string Join(string separator, IEnumerable<string> values);

}

It concatenates the string values with a single separator between each 2 adjacent string values. Similarly, a general ConcatJoin query can be defined as:

public static IEnumerable<TSource> ConcatJoin<TSource>(

this IEnumerable<TSource> source, TSource separator)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (iterator.MoveNext())

yield return iterator.Current; // Deferred execution.

while (iterator.MoveNext())

yield return separator; // Deferred execution.

yield return iterator.Current; // Deferred execution.

}

The built-in Append/Prepend can append/prepend 1 value to the source sequence. So the following overloads can be defined to support multiple values:

public static IEnumerable<TSource> Append<TSource>(

this IEnumerable<TSource> source, params TSource[] values) =>

source.Concat(values);

public static IEnumerable<TSource> Prepend<TSource>(

this IEnumerable<TSource> source, params TSource[] values) =>

values.Concat(source);

The following AppendTo/PrependTo extension method are defined for single value, which canto make code more fluent:

public static IEnumerable<TSource> AppendTo<TSource>(

this TSource value, IEnumerable<TSource> source) =>

source.Append(value);

public static IEnumerable<TSource> PrependTo<TSource>(

this TSource value, IEnumerable<TSource> source) =>

source.Prepend(value);

Partitioning

Similar to string.Substring, a general Subsequence query can be defined as:

public static IEnumerable<TSource> Subsequence<TSource>(

this IEnumerable<TSource> source, int startIndex, int count) =>

source.Skip(startIndex).Take(count);

The following Pagination query is useful to paginate a sequence of values:

public static IEnumerable<TSource> Pagination<TSource>(

this IEnumerable<TSource> source, int pageIndex, int countPerPage) =>

source.Skip(pageIndex * countPerPage).Take(countPerPage);

Ordering

In LINQ to Objects, the ordering queries must compare objects to determine their order, so they all have overload to accept IComparer<T> parameter. This interface can be viewed as a wrapper of a simple comparison functions:

namespace System.Collections.Generic

public interface IComparer<in T>

int Compare(T x, T y);

public interface IEqualityComparer<in T>

bool Equals(T x, T y);

int GetHashCode(T obj);

}

In C#, interfaces are less convenient then functions. C# supports lambda expression to define anonymous functions inline, but does not support anonymous class to enable inline interface. For the LINQ queries accepting interface parameter, they are easier to be called if they can accept function parameter instead. To Implement this, the following ToComparer function can be defined to convert a compare functions to an IComparer<T> interface:

private static IComparer<T> ToComparer<T>(Func<T, T, int> compare) =>

Comparer<T>.Create(new Comparison<T>(compare));

It simply calls a .NET Standard built-in API Comparer<T>.Create for the IComparer<T> instantiation. Now the ordering queries’ overloads can be defined as a higher-order functions to accept a (T, T) –> int function instead of IComparer<T> interface:

public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TKey, TKey, int>compare) =>

source.OrderBy(keySelector, ToComparer(compare));

public static IOrderedEnumerable<TSource> OrderByDescending<TSource, TKey>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TKey, TKey, int>compare) =>

source.OrderByDescending(keySelector, ToComparer(compare));

public static IOrderedEnumerable<TSource> ThenBy<TSource, TKey>(

this IOrderedEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TKey, TKey, int>compare) =>

source.ThenBy(keySelector, ToComparer(compare));

public static IOrderedEnumerable<TSource> ThenByDescending<TSource, TKey>(

this IOrderedEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TKey, TKey, int>compare) =>

source.ThenByDescending(keySelector, ToComparer(compare));

Grouping, join, and set

In LINQ to Objects, there are also queries need to compare objects’ equality to determine the grouping, join, and set operation, so they all have overload to accept IEqualityComparer<T> parameter. .NET Standard does not provide a built-in API for IEqualityComparer<T> instantiation from functions (F# core library provides a Microsoft.FSharp.Collections.HashIdentity type to wrap functions for IEqualityComparer<T>, but it is not easy to use in C#). So first, a EqualityComparerWrapper<T> type can be defined to implement IEqualityComparer<T>, then a higher-order function ToEqualityComparer can be defined to convert a equals functions and a getHashCode function to an IEqualityComparer<T> interface:

internal class EqualityComparerWrapper<T> : IEqualityComparer<T>

private readonly Func<T, T, bool> equals;

private readonly Func<T, int> getHashCode;

public EqualityComparerWrapper(Func<T, T, bool> equals, Func<T, int> getHashCode = null) =>

(this.equals, this.getHashCode) = (@equals, getHashCode ?? (value => value.GetHashCode()));

public bool Equals(T x, T y) => this.equals(x, y);

public int GetHashCode(T obj) => this.getHashCode(obj);

private static IEqualityComparer<T> ToEqualityComparer<T>(

Func<T, T, bool> equals, Func<T, int> getHashCode = null) =>

new EqualityComparerWrapper<T>(equals, getHashCode);

The getHashCode function is optional, because any type already inherits a GetHashCode method from object. Similar to ordering queries, the following functional overloads can be defined for GroupBy, Join, GroupJoin, Distinct, Union, Intersect, Except:

public static IEnumerable<TResult> GroupBy<TSource, TKey, TElement, TResult>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

Func<TKey, IEnumerable<TElement>, TResult> resultSelector,

Func<TKey, TKey, bool>equals,

Func<TKey, int> getHashCode = null) =>

source.GroupBy(keySelector, elementSelector, resultSelector, ToEqualityComparer(equals, getHashCode));

public static IEnumerable<TResult> Join<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, TInner, TResult>resultSelector,

Func<TKey, TKey, bool>equals,

Func<TKey, int> getHashCode = null) =>

outer.Join(

inner,

outerKeySelector,

innerKeySelector,

resultSelector,

ToEqualityComparer(equals, getHashCode));

public static IEnumerable<TResult> GroupJoin<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, IEnumerable<TInner>, TResult> resultSelector,

Func<TKey, TKey, bool>equals,

Func<TKey, int> getHashCode = null) =>

outer.GroupJoin(

inner,

outerKeySelector,

innerKeySelector,

resultSelector,

ToEqualityComparer(equals, getHashCode));

public static IEnumerable<TSource> Distinct<TSource>(

this IEnumerable<TSource> source,

Func<TSource, TSource, bool>equals,

Func<TSource, int> getHashCode = null) =>

source.Distinct(ToEqualityComparer(equals, getHashCode));

public static IEnumerable<TSource> Union<TSource>(

this IEnumerable<TSource> first,

IEnumerable<TSource>second,

Func<TSource, TSource, bool>equals,

Func<TSource, int> getHashCode = null) =>

first.Union(second, ToEqualityComparer(equals, getHashCode));

public static IEnumerable<TSource> Intersect<TSource>(

this IEnumerable<TSource> first,

IEnumerable<TSource>second,

Func<TSource, TSource, bool>equals,

Func<TSource, int> getHashCode = null) =>

first.Intersect(second, ToEqualityComparer(equals, getHashCode));

public static IEnumerable<TSource> Except<TSource>(

this IEnumerable<TSource> first,

IEnumerable<TSource>second,

Func<TSource, TSource, bool>equals,

Func<TSource, int> getHashCode = null) =>

first.Except(second, ToEqualityComparer(equals, getHashCode));

List

The List<T> type provides handy methods, which can be implemented for sequence too. The following Insert query is similar to List<T>.Insert, it outputs a new sequence with the specified value is inserted at the specified index:

public static IEnumerable<TSource> Insert<TSource>(

this IEnumerable<TSource> source, int index, TSource value)

if (index< 0)

throw new ArgumentOutOfRangeException(nameof(index));

IEnumerable<TSource> InsertGenerator()

int currentIndex = 0;

foreach (TSource sourceValue in source)

if (currentIndex == index)

yield return value; // Deferred execution.

yield return sourceValue; // Deferred execution.

currentIndex = checked(currentIndex + 1);

if (index == currentIndex)

yield return value; // Deferred execution.

else if (index> currentIndex)

throw new ArgumentOutOfRangeException(

nameof(index),

$"{nameof(index)} must be within the bounds of {nameof(source)}.");

return InsertGenerator();

}

The above Insert query is more functional than List<T>.Insert. List<T>.Insert has no output, so it is not fluent and it implements immediate execution. It is also impure by mutating the list in place. The above Insert query follows the iterator pattern, and uses yield statement to implement deferred execution. It outputs a new sequence, so it is fluent, and it is a pure function since it does not mutate the source sequence.

RemoveAt outputs a new sequence with a value removed at the specified index:

public static IEnumerable<TSource> RemoveAt<TSource>(

this IEnumerable<TSource> source, int index)

if (index< 0)

throw new ArgumentOutOfRangeException(nameof(index));

IEnumerable<TSource> RemoveAtGenerator()

int currentIndex = 0;

foreach (TSource value in source)

if (currentIndex != index)

yield return value; // Deferred execution.

currentIndex = checked(currentIndex + 1);

if (index> = currentIndex)

throw new ArgumentOutOfRangeException(nameof(index));

return RemoveAtGenerator();

}

Remove outputs a new sequence with the first occurrence of the specified value removed. Besides being deferred and lazy, it also accepts an optional equality comparer:

public static IEnumerable<TSource> Remove<TSource>(

this IEnumerable<TSource>source,

TSource value,

IEqualityComparer<TSource> comparer = null)

comparer = comparer ?? EqualityComparer<TSource>.Default;

bool isRemoved = false;

foreach (TSource sourceValue in source)

if (!isRemoved&& comparer.Equals(sourceValue, value))

isRemoved = true;

else

yield return sourceValue; // Deferred execution.

}

RemoveAll outputs a new sequence with all occurrences of the specified value removed:

public static IEnumerable<TSource> RemoveAll<TSource>(

this IEnumerable<TSource>source,

TSource value,

IEqualityComparer<TSource> comparer = null)

comparer = comparer ?? EqualityComparer<TSource>.Default;

foreach (TSource sourceValue in source)

if (!comparer.Equals(sourceValue, value))

yield return sourceValue; // Deferred execution.

}

Since Remove and RemoveAll tests the equality of objects to determine which objects to remove, the following higher-order function overloads can be defined for convenience:

public static IEnumerable<TSource> Remove<TSource>(

this IEnumerable<TSource> source,

TSource value,

Func<TSource, TSource, bool> equals,

Func<TSource, int> getHashCode = null) =>

source.Remove(value, ToEqualityComparer(@equals, getHashCode));

public static IEnumerable<TSource> RemoveAll<TSource>(

this IEnumerable<TSource> source,

TSource value,

Func<TSource, TSource, bool> equals,

Func<TSource, int> getHashCode = null) =>

source.RemoveAll(value, ToEqualityComparer(@equals, getHashCode));

Collection queries

Conversion

ToDictionary and ToLookup accept IEqualityComparer<T> parameter to test the equality of keys. Their functional overloads can be defined:

public static Dictionary<TKey, TElement> ToDictionary<TSource, TKey, TElement>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

Func<TKey, TKey, bool>equals,

Func<TKey, int> getHashCode = null) =>

source.ToDictionary(keySelector, elementSelector, ToEqualityComparer(equals, getHashCode));

public static ILookup<TKey, TElement> ToLookup<TSource, TKey, TElement>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

Func<TKey, TKey, bool>equals,

Func<TKey, int> getHashCode = null) =>

source.ToLookup(keySelector, elementSelector, ToEqualityComparer(equals, getHashCode));

Value queries

Aggregation

.NET provides basic aggregation queries, including Sum/Average/Max/Min queries. In reality, it is also common to calculate the variance, standard deviation, and percentile. The following VariancePopulation/VarianceSample/Variance queries are equivalent to Excel VAR.P/VAR.S/VAR functions:

public static double VariancePopulation<TSource, TKey>( // Excel VAR.P function.

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IFormatProvider formatProvider = null)

where TKey : IConvertible

double[] keys = source.Select(key => keySelector(key).ToDouble(formatProvider)).ToArray();

double mean = keys.Average();

return keys.Sum(key => (key - mean) * (key - mean)) / keys.Length;

public static double VarianceSample<TSource, TKey>( // Excel VAR.S function.

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IFormatProvider formatProvider = null)

where TKey : IConvertible

double[] keys = source.Select(key => keySelector(key).ToDouble(formatProvider)).ToArray();

double mean = keys.Average();

return keys.Sum(key => (key - mean) * (key - mean)) / (keys.Length - 1);

public static double Variance<TSource, TKey>( // Excel VAR function.

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IFormatProvider formatProvider = null)

where TKey : IConvertible =>

source.VarianceSample(keySelector, formatProvider);

And the following StandardDeviationPopulation/StabdardDeviationSample/StabdardDeviation queries implements Excel STDEV.P/STDEV.S/STDEV functions:

public static double StandardDeviationPopulation<TSource, TKey>( // Excel STDEV.P function.

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IFormatProvider formatProvider = null)

where TKey : IConvertible =>

Math.Sqrt(source.VariancePopulation(keySelector, formatProvider));

public static double StandardDeviationSample<TSource, TKey>( // Excel STDEV.S function.

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IFormatProvider formatProvider = null)

where TKey : IConvertible =>

Math.Sqrt(source.VarianceSample(keySelector, formatProvider));

public static double StandardDeviation<TSource, TKey>( // Excel STDEV function.

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IFormatProvider formatProvider = null)

where TKey : IConvertible =>

Math.Sqrt(source.Variance(keySelector, formatProvider));

And the following PercentileExclusive/PercentileInclusive/Percentile implement Excel PERCENTILE.EXC/PERCENTILE.INC/PERCENTILE functions:

public static double PercentileExclusive<TSource, TKey>( // Excel PERCENTILE.EXC function.

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

double percentile,

IComparer<TKey> comparer = null,

IFormatProvider formatProvider = null)

where TKey : IConvertible

if (percentile < 0 || percentile > 1)

throw new ArgumentOutOfRangeException(nameof(percentile), $"{nameof(percentile)} must be between 0 and 1.");

comparer = comparer ?? Comparer<TKey>.Default;

TKey[] orderedKeys = source.Select(keySelector).OrderBy(key => key, comparer).ToArray();

int length = orderedKeys.Length;

if (percentile < (double)1 / length || percentile > 1 - (double)1 / (length + 1))

throw new ArgumentOutOfRangeException(

nameof(percentile),

$"{nameof(percentile)} must be in the range between (1 / source.Count()) and (1 - 1 / source.Count()).");

double index = percentile * (length + 1) - 1;

int integerComponentOfIndex = (int)index;

double decimalComponentOfIndex = index - integerComponentOfIndex;

double keyAtIndex = orderedKeys[integerComponentOfIndex].ToDouble(formatProvider);

double keyAtNextIndex = orderedKeys[integerComponentOfIndex + 1].ToDouble(formatProvider);

return keyAtIndex + (keyAtNextIndex - keyAtIndex) * decimalComponentOfIndex;

public static double PercentileInclusive<TSource, TKey>( // Excel PERCENTILE.INC function.

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

double percentile,

IComparer<TKey> comparer = null,

IFormatProvider formatProvider = null)

where TKey : IConvertible

if (percentile < 0 || percentile > 1)

throw new ArgumentOutOfRangeException(nameof(percentile), $"{nameof(percentile)} must be between 0 and 1.");

comparer = comparer ?? Comparer<TKey>.Default;

TKey[] orderedKeys = source.Select(keySelector).OrderBy(key => key, comparer).ToArray();

int length = orderedKeys.Length;

double index = percentile * (length - 1);

int integerComponentOfIndex = (int)index;

double decimalComponentOfIndex = index - integerComponentOfIndex;

double keyAtIndex = orderedKeys[integerComponentOfIndex].ToDouble(formatProvider);

if (integerComponentOfIndex >= length - 1)

return keyAtIndex;

double keyAtNextIndex = orderedKeys[integerComponentOfIndex + 1].ToDouble(formatProvider);

return keyAtIndex + (keyAtNextIndex - keyAtIndex) * decimalComponentOfIndex;

public static double Percentile<TSource, TKey>( // Excel PERCENTILE function.

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

double percentile,

IComparer<TKey> comparer = null,

IFormatProvider formatProvider = null)

where TKey : IConvertible

if (percentile < 0 || percentile > 1)

throw new ArgumentOutOfRangeException(nameof(percentile), $"{nameof(percentile)} must be between 0 and 1.");

return PercentileInclusive(source, keySelector, percentile, comparer, formatProvider);

}

Quantifiers

string has a very useful IsNullOrEmpty method, and here is the LINQ version:

public static bool IsNullOrEmpty<TSource>(this IEnumerable<TSource> source) =>

source == null || !source.Any();

Contains compares the objects to determine the existance, so it can accept IEqualityComparer<T> parameter. It can be overloaded with functions for convenience:

public static bool Contains<TSource>(

this IEnumerable<TSource>source,

TSource value,

Func<TSource, TSource, bool> equals,

Func<TSource, int> getHashCode = null) =>

source.Contains(value, ToEqualityComparer(equals, getHashCode));

Equality

SequentialEqual compares the objects as well, so it also accepts IEqualityComparer<T>. It can be overloaded with functions:

public static bool SequenceEqual<TSource>(

this IEnumerable<TSource> first,

IEnumerable<TSource>second,

Func<TSource, TSource, bool>equals,

Func<TSource, int> getHashCode = null) =>

first.SequenceEqual(second, ToEqualityComparer(equals, getHashCode));

List

IndexOf is similar to List<T>.IndexOf. It finds the index of first occurrence of the specified value. –1 is returned if the specified value is not found:

public static int IndexOf<TSource>(

this IEnumerable<TSource>source,

TSource value,

IEqualityComparer<TSource> comparer = null)

comparer = comparer ?? EqualityComparer<TSource>.Default;

int index = 0;

foreach (TSource sourceValue in source)

if (comparer.Equals(sourceValue, value))

return index;

index = checked(index + 1);

return -1;

}

LastIndexOf is similar to List<T>.LastIndexOf. It finds the index of last occurrence of the specified value:

public static int LastIndexOf<TSource>(

this IEnumerable<TSource>source,

TSource value,

IEqualityComparer<TSource> comparer = null)

comparer = comparer ?? EqualityComparer<TSource>.Default;

int lastIndex = -1;

int index = 0;

foreach (TSource sourceValue in source)

if (comparer.Equals(sourceValue, value))

lastIndex = index;

index = checked(index + 1);

return lastIndex;

}

Again, here are the functional overloads of IndexOf and LastIndexOf:

public static int IndexOf<TSource>(

this IEnumerable<TSource> source,

TSource value,

Func<TSource, TSource, bool> equals,

Func<TSource, int> getHashCode = null) =>

source.IndexOf(value, ToEqualityComparer(equals, getHashCode));

public static int LastIndexOf<TSource>(

this IEnumerable<TSource> source,

TSource value,

Func<TSource, TSource, bool> equals,

Func<TSource, int> getHashCode = null) =>

source.LastIndexOf(value, ToEqualityComparer(equals, getHashCode));

Void queries

Iteration

EnumerableEx.ForEach from Ix is very handy. It can fluently execute the query and process the results. It works like foreach statement, but it does not support breaking the iterations like the break statement in foreach statement. So here is an improved EnumerableX.ForEach, with a slightly different callback function:

public static void ForEach<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> onNext)

foreach (TSource value in source)

if (!onNext(value))

break;

}

The callback function is of type TSource -> bool. When its output is true, the iteration continues; when its output is false, ForEach stops execution. And the indexed overload is:

public static void ForEach<TSource>(

this IEnumerable<TSource> source, Func<TSource, int, bool> onNext)

int index = 0;

foreach (TSource value in source)

if (!onNext(value, index))

break;

index = checked(index + 1);

}

The last overload does not accept the callback function. It just iterates the source sequence:

public static void ForEach(this IEnumerable source)

IEnumerator iterator = source.GetEnumerator();

try

while (iterator.MoveNext()) { }

finally

(iterator as IDisposable)?.Dispose();

}

It can be used to just execute a LINQ query and ignore all query results.

Summary

This chapter demonstrates how to implement custom LINQ to Objects queries, including generation queries, list-API-like queries, aggregation queries to compute variance, standard deviation, and percentile, and also functional overloads for the standard ordering, grouping, join, set, conversion, quantifier, and equality queries that compares objects, and many more.

LINQ to Objects in Depth (6) Advanced Queries in Interactive Extensions (Ix)

Sat, 06 Jul 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to Objects in Depth series]

The previous 2 chapters discussed the LINQ to Objects standard queries. Besides these built-in queries provided by System.Linq.Enumerable type in .NET Standard, Microsoft also provides additional LINQ to Objects queries through the System.Interactive NuGet package (aka Interactive Extensions library, or Ix). Ix has a System.Linq.EnumerableEx type with the following queries:

· Sequence queries: output a new IEnumerable<T> sequence (deferred execution)

o Generation: Defer, Create, Return, Repeat

o Filtering: IgnoreElements*, DistinctUntilChanged

o Mapping: SelectMany, Scan, Expand

o Concatenation: Concat, StartWith

o Set: Distinct

o Partitioning: TakeLast*, SkipLast**

o Conversion: Hide

o Buffering: Buffer*, Share, Publish, Memoize

o Exception handling: Throw, Catch, Finally, OnErrorResumeNext, Retry

o Control flow: If, Case, Using, While, DoWhile, Generate, For

o Iteration: Do

· Value queries: output a single value (immediate execution)

o Aggregation: Min, Max, MinBy, MaxBy

o Quantifiers: isEmpty

· Void queries: no output (immediate execution)

o Iteration: ForEach

Many of these queries are handy and useful. However, there is not much documentation provided from Microsoft, except the APIs’ XML comments. This chapter discusses these queries by either providing examples and/or demonstrating their internal implementation, whichever is more intuitive.

Similar to Enumerable queries, the EnumerableEx queries with a sequence output implements deferred execution, and the other queries implements immediate execution. For the sequence queries, the ones marked with * implement eager evaluation, and the unmarked queries implements lazy evaluation. The SkipLast query marked with ** is slightly different, it can be fully eager evaluation or partially eager evaluation, which is discussed later.

Sequence queries

Similar to the standard sequence queries, the Ix sequence queries follow iterator pattern to implement deferred execution. Many of them uses yield statement for generator, and some queries are implemented by the composition of other standard and Ix queries.

Generation

Defer accepts a sequence factory function:

public static IEnumerable<TResult> Defer<TResult>(

Func<IEnumerable<TResult>> enumerableFactory)

foreach (TResult value in enumerableFactory())

yield return value; // Deferred execution.

}

And it defers the execution of the factory function:

internal static void Defer(IEnumerable<string> source)

IEnumerable<string> Distinct()

"Instantiate hash set.".WriteLine();

HashSet<string> hashSet = new HashSet<string>();

return source.Where(hashSet.Add); // Deferred execution.

IEnumerable<string> distinct1 = Distinct() // Hash set is instantiated.

.Where(@string => @string.Length > 10);

IEnumerable<string> distinct2 = EnumerableEx.Defer(Distinct) // Hash set is not instantiated.

.Where(@string => @string.Length > 10);

}

Similarly, Create accepts an iterator factory function, and delays its execution:

public static IEnumerable<TResult> Create<TResult>(

Func<IEnumerator<TResult>> getEnumerator)

using (IEnumerator<TResult> iterator = getEnumerator())

while (iterator.MoveNext())

yield return iterator.Current; // Deferred execution.

}

The other overload of Create is not so intuitive:

public static IEnumerable<T> Create<T>(Action<IYielder<T>> create);

It accepts a callback function of type System.Linq.IYielder<T> –> void. IYielder<T> has 2 methods, Return and Break, representing the 2 forms of yield statement.

public interface IYielder<in T>

IAwaitable Return(T value);

IAwaitable Break();

}

In C#, lambda expression does not support yield statements, compiling the following code causes error CS1621: The yield statement cannot be used inside an anonymous method or lambda expression.

// Cannot be compiled.

internal static void Create()

Func<IEnumerable<int>> sequenceFactory = () =>

yield return 0;

yield return 1;

yield break;

yield return 2;

};

IEnumerable<int> sequence = sequenceFactory();

sequence.WriteLines(); // 0 1

}

Here Create provides a way to virtually use the yield statements in lambda expression:

internal static void Create()

Action<IYielder<int>> sequenceFactory = async yield =>

await yield.Return(0); // yield return 0;

await yield.Return(1); // yield return 1;

await yield.Break(); // yield break;

await yield.Return(2); // yield return 2;

};

IEnumerable<int>sequence = EnumerableEx.Create(sequenceFactory);

sequence.WriteLines(); // 0 1

}

IYielder<T> is a good invention before C# 7.0 introduces local function, but at runtime, it can have unexpected iterator behaviour when used with more complex control flow, like try-catch statement. Please avoid using this query. In the above examples, define local function to use yield return statement:

internal static void Create()

IEnumerable<int>SequenceFactory()

yield return 0; // Deferred execution.

yield return 1;

yield break;

yield return 2;

IEnumerable<int>sequence = SequenceFactory();

sequence.WriteLines(); // 0 1

}

Return just wraps value in a singleton sequence:

public static IEnumerable<TResult> Return<TResult>(TResult value)

yield return value; // Deferred execution.

}

It is called Return, because “return” is a term used in functional languages like Haskell, which means to wrap something in a monad (Monad is discussed in detail in the Category Theory chapters). However, in C# “return” means the current function member gives control to its caller with an optional output. It could be more consistent with .NET naming convention if this function is named as FromValue, similar to Task.FromResult, Task.FromException, DateTime.FromBinary, DateTimeOffset.FromFileTime, TimeSpan.FromSeconds, RegistryKey.FromHandle, etc.

Repeat generates an infinite sequence by repeating a value forever:

public static IEnumerable<TResult> Repeat<TResult>(TResult value)

while (true)

yield return value; // Deferred execution.

}

Another overload repeats values in the specified sequence. Its implementation is equivalent to:

public static IEnumerable<TSource> Repeat<TSource>(this IEnumerable<TSource> source, int? count = null)

if (count == null)

while (true)

foreach (TSource value in source)

yield return value; // Deferred execution.

for (int i = 0; i< count; i++)

foreach (TSource value in source)

yield return value; // Deferred execution.

}

When count is not provided, it repeats the values of the source sequence forever.

Filtering

IgnoreElements filters out all values from the source sequence:

public static IEnumerable<TSource> IgnoreElements<TSource>(this IEnumerable<TSource> source)

foreach (TSource value in source) { } // Eager evaluation.

yield break; // Deferred execution.

}

DistinctUntilChanged removes the continuous duplication:

public static IEnumerable<TSource> DistinctUntilChanged<TSource>(this IEnumerable<TSource> source);

public static IEnumerable<TSource> DistinctUntilChanged<TSource>(

this IEnumerable<TSource> source, IEqualityComparer<TSource> comparer);

public static IEnumerable<TSource> DistinctUntilChanged<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IEnumerable<TSource> DistinctUntilChanged<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector, IEqualityComparer<TKey> comparer);

For example:

internal static void DistinctUntilChanged()

IEnumerable<int>source = new int[]

0, 0, 0, /* Change. */ 1, 1, /* Change. */ 0, 0, /* Change. */ 2, /* Change. */ 1, 1

};

source.DistinctUntilChanged().WriteLines(); // 0 1 0 2 1

}

Mapping

A SelectMany overload is provided to map source sequence’s each value to the other sequence:

public static IEnumerable<TOther> SelectMany<TSource, TOther>(

this IEnumerable<TSource> source, IEnumerable<TOther> other) =>

source.SelectMany(value => other);

Scan accepts the same parameters as Aggregate. The difference is, Aggregate outputs one final accumulation step’s result, Scan returns a sequence of all accumulation steps’ results. Its implementation is equivalent to:

public static IEnumerable<TSource> Scan<TSource>(

this IEnumerable<TSource> source, Func<TSource, TSource, TSource> func)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (!iterator.MoveNext())

yield break; // Deferred execution.

TSource accumulate = iterator.Current;

while (iterator.MoveNext())

yield return accumulate = func(accumulate, iterator.Current); // Deferred execution.

public static IEnumerable<TAccumulate> Scan<TSource, TAccumulate>(

this IEnumerable<TSource> source, TAccumulate seed, Func<TAccumulate, TSource, TAccumulate> func) =>

source.Select(value => seed = func(seed, value));

For example:

internal static void Scan()

int finalProduct = Int32Source().Aggregate((product, int32) => product * int32).WriteLine();

// ((((-1 * 1) * 2) * 3) * -4) => 24.

IEnumerable<int> allProducts = Int32Source().Scan((product, int32) => product * int32).WriteLines();

// ((((-1 * 1) * 2) * 3) * -4) => { -1, -2, -6, 24 }.

}

Expand maps source values with the selector, then maps the result values with the selector, and keeps going on.

public static IEnumerable<TSource> Expand<TSource>(this IEnumerable<TSource> source, Func<TSource, IEnumerable<TSource>> selector);

In the following example, selector maps each value to a singleton sequence:

internal static void ExpandSingle()

Enumerable

.Range(0, 5)

.Expand(int32 => EnumerableEx.Return(int32 * int32))

.Take(25)

.WriteLines();

// 0 1 2 3 4, map each int32 to { int32 * int32 } =>

// 0 1 4 9 16, map each int32 to { int32 * int32 }: =>

// 0 1 16 81 256, map each int32 to { int32 * int32 } =>

// 0 1 256 6561 65536, map each int32 to { int32 * int32 } =>

// 0 1 65536 43046721 4294967296, ...

}

The mapping can go on forever and results an infinite sequence. If selector maps each value to a sequence with more than one values, then the result sequences grows rapidly:

internal static void ExpandMuliple()

Enumerable

.Range(0, 5)

.Expand(int32 => Enumerable.Repeat(int32, 2))

.Take(75)

.WriteLines();

// 0 1 2 3 4 => map each int32 to { int32, int32 }:

// 0 0 1 1 2 2 3 3 4 4 => map each int32 to { int32, int32 }:

// 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4 => map each int32 to { int32, int32 }:

// 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 => ...

}

If selector maps each value to empty sequence, the expanding ends after all source values are iterated:

internal static void ExpandNone()

Enumerable

.Range(0, 5)

.Expand(int32 => Enumerable.Empty<int>())

.Take(100)

.WriteLines();

// 0 1 2 3 4 => map each int32 to { }.

}

Concatenation

2 more overloads of Concat is provided to concatenate any number of sequences:

public static IEnumerable<TSource> Concat<TSource>(

this IEnumerable<IEnumerable<TSource>> sources) =>

sources.SelectMany(source => source);

public static IEnumerable<TSource> Concat<TSource>(

params IEnumerable<TSource>[] sources) => sources.Concat();

By concatenating the sequences one after another, Concat flattens a hierarchical 2-level-sequence into a flat 1-level-sequence, which works the same as SelectMany.

StartWith prepend the specified values to the source sequence. It is similar to Prepend. Prepend accepts a single prefix value, but StartWith supports multiple prefix values:

public static IEnumerable<TSource> StartWith<TSource>(

this IEnumerable<TSource> source, params TSource[] values) => values.Concat(source);

Set

An overload of Distinct is provided to accept a key selector function:

public static IEnumerable<TSource> Distinct<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector, IEqualityComparer<TKey> comparer = null)

HashSet<TKey>hashSet = new HashSet<TKey>(comparer);

foreach (TSource value in source)

if (hashSet.Add(keySelector(value)))

yield return value; // Deferred execution.

}

Partitioning

Skip/Take skips/takes the specified number of values at the beginning of the source sequence. In contrast, SkipLast/TakeLast skips/takes the specified number of values at the end of the source sequence:

public static IEnumerable<TSource> SkipLast<TSource>(this IEnumerable<TSource> source, int count);

public static IEnumerable<TSource> TakeLast<TSource>(this IEnumerable<TSource> source, int count);

For example:

internal static void SkipLastTakeLast()

int[] skipFirst2 = Enumerable.Range(0, 5).Skip(2).ToArray(); // 2 3 4.

int[] skipLast2 = Enumerable.Range(0, 5).SkipLast(2).ToArray(); // 0 1 2.

int[] takeFirst2 = Enumerable.Range(0, 5).Take(2).ToArray(); // 0 1.

int[] takeLast2 = Enumerable.Range(0, 5).TakeLast(2).ToArray(); // 3 4.

}

The implementation of SkipLast/TakeLast is very interesting. As already discussed, Take implements lazy evaluation. However, TakeLast has to pull all values to know which are the tail values of the source sequence. So TakeLast implements eager evaluation, and uses a queue to store the tail values:

public static IEnumerable<TSource> TakeLast<TSource>(this IEnumerable<TSource> source, int count)

if (count < 0)

throw new ArgumentOutOfRangeException(nameof(count));

IEnumerable<TSource> TakeLastGGenerator()

if (count <= 0)

yield break; // Deferred execution.

Queue<TSource>lastValues = new Queue<TSource>(count);

foreach (TSource value in source)

if (lastValues.Count >= count)

lastValues.Dequeue();

lastValues.Enqueue(value);

} // Eager evaluation.

while (lastValues.Count> 0)

yield return lastValues.Dequeue(); // Deferred execution.

return TakeLastGGenerator();

}

SkipLast also uses a queue to store the tail values:

public static IEnumerable<TSource> SkipLast<TSource>(this IEnumerable<TSource> source, int count)

if (count < 0)

throw new ArgumentOutOfRangeException(nameof(count));

IEnumerable<TSource> SkipLastGenerator()

Queue<TSource>lastValues = new Queue<TSource>();

foreach (TSource value in source)

lastValues.Enqueue(value);

if (lastValues.Count > count) // Can be lazy, eager, or between.

yield return lastValues.Dequeue(); // Deferred execution.

return SkipLastGenerator();

}

It uses count as the max length of the queue. When SkipLast starts to execute, it evaluates values to fill the queue. When the queue is full, each new value is enqueued, and the head value of the queue is dequeued and yielded. So, at the end of query execution, the values still stored in the queue are exactly the last values to skip. If count is equal to or greater than the source sequence’s value count, when executing query, all values are pulled from the source sequence and stored in the queue, and nothing is yielded to the caller, which is fully eager evaluation similar to IgnoreElements. If count is less than the source’s value count, when executing query, some values are pulled from the source sequence to fill the queue, then values are yielded, which can be viewed as partially eager evaluation. When count is 0, it does not skip anything, just simply yield each source value, which is like lazy evaluation. So SkipLast’s eagerness/laziness depends on the count of values to skip.

Conversion

Hide has the same signature as AsEnumerable. As previously demonstrated, AsEnumerable simply outputs the source sequence itself to caller. Hide returns a new generator to hide the source sequence from the caller:

public static IEnumerable<TSource> Hide<TSource>(this IEnumerable<TSource> source)

foreach (TSource value in source)

yield return value; // Deferred execution.

}

The difference is, the output sequence of AsEnumerable can be converted back to the original type, which the output sequence of Hide cannot, since it is a newly constructed generator:

internal static void Hide()

List<int>source = new List<int>() { 1, 2 };

IEnumerable<int>readWrite = source.AsEnumerable();

object.ReferenceEquals(source, readWrite).WriteLine(); // True

((List<int>)readWrite).Reverse(); // List<T>.Reverse.

((List<int>)readWrite).Add(3); // List<T>.Add.

IEnumerable<int> readOnly = source.Hide();

object.ReferenceEquals(source, readOnly).WriteLine(); // False

}

Buffering

Buffer segments the source sequence into smaller lists:

public static IEnumerable<IList<TSource>> Buffer<TSource>(this IEnumerable<TSource> source, int count, int skip);

Here count is the length of each smaller list, and skip is the offset to start the next list. For example:

internal static void Buffer()

IEnumerable<IList<int>> buffers1 = Enumerable.Range(0, 5).Buffer(2, 1);

// {

// { 0, 1 }, { 1, 2 }, { 2, 3 }, { 3, 4 }, { 4 }

// }

IEnumerable<IList<int>> buffers2 = Enumerable.Range(0, 5).Buffer(2, 2); // Equivalent to Buffer(2).

// {

// { 0, 1 }, { 2, 3 }, { 4 }

// }

IEnumerable<IList<int>> buffers3 = Enumerable.Range(0, 5).Buffer(2, 3);

// {

// { 0, 1 }, { 3, 4 }

// }

}

Buffer implements eager evaluation. it creates all the smaller lists when the first list is pulled.

The other overload without skip uses count as skip:

public static IEnumerable<IList<TSource>> Buffer<TSource>(this IEnumerable<TSource> source, int count);

In above example, calling Buffer(2, 2) is equivalent to Buffer(2).

Share buffers the values of a sequence and share them with several iterators:

public static IBuffer<TSource> Share<TSource>(this IEnumerable<TSource> source);

The output type System.Linq.IBuffer<T> is a composition of IEnumerable<T> and IDisposable:

namespace System.Linq

public interface IBuffer<out T> : IEnumerable<T>, IEnumerable, IDisposable { }

}

By default, an IEnumerable<T> sequence’s multiple iterators are independent from each other. When these iterators are called, callers pull independent values from each iterator. In contrast, multiple shared iterators work as if they are the same single iterator:

internal static void Share()

IEnumerable<int>sequence = Enumerable.Range(0, 5);

IEnumerator<int>independentIteratorA = sequence.GetEnumerator();

IEnumerator<int>independentIteratorB = sequence.GetEnumerator(); // A|B|C

independentIteratorA.MoveNext(); independentIteratorA.Current.WriteLine(); // 0| |

independentIteratorB.MoveNext(); independentIteratorB.Current.WriteLine(); // |0|

independentIteratorA.MoveNext(); independentIteratorA.Current.WriteLine(); // 1| |

IEnumerator<int> independentIteratorC = sequence.GetEnumerator(); // | |

independentIteratorC.MoveNext(); independentIteratorC.Current.WriteLine(); // | |0

independentIteratorA.MoveNext(); independentIteratorA.Current.WriteLine(); // 2| |

independentIteratorB.MoveNext(); independentIteratorB.Current.WriteLine(); // |1|

independentIteratorA.MoveNext(); independentIteratorA.Current.WriteLine(); // 3| |

// ...

IBuffer<int> share = Enumerable.Range(0, 5).Share();

IEnumerator<int>sharedIterator1 = share.GetEnumerator();

IEnumerator<int>sharedIterator2 = share.GetEnumerator(); // A|B|C

sharedIterator1.MoveNext(); sharedIterator1.Current.WriteLine(); // 0| |

sharedIterator2.MoveNext(); sharedIterator2.Current.WriteLine(); // |1|

sharedIterator1.MoveNext(); sharedIterator1.Current.WriteLine(); // 2| |

IEnumerator<int> sharedIterator3 = share.GetEnumerator(); // | |

sharedIterator3.MoveNext(); sharedIterator3.Current.WriteLine(); // | |3

share.Dispose();

sharedIterator1.MoveNext(); // ObjectDisposedException.

sharedIterator2.MoveNext(); // ObjectDisposedException.

sharedIterator3.MoveNext(); // ObjectDisposedException.

}

When pulling values with multiple independent iterators, each value can be pulled multiple times. When pulling values with multiple shared iterators, each value can only be pulled once. And IBuffer<T>.Dispose terminates the sharing. After calling Dispose, all shared iterators’ MoveNext throws ObjectDisposedException.

The other overload accepts a selector function:

public static IEnumerable<TResult> Share<TSource, TResult>(

this IEnumerable<TSource> source,

Func<IEnumerable<TSource>, IEnumerable<TResult>> selector) =>

Create(() => selector(source.Share()).GetEnumerator());

For example:

internal static void ConcatShared()

IEnumerable<int>source1 = Enumerable.Range(0, 5);

source1.Concat(source1).WriteLines(); // 0 1 2 3 4 0 1 2 3 4

using (IBuffer<int> source2 = Enumerable.Range(0, 5).Share())

source2.Concat(source2).WriteLines(); // 0 1 2 3 4

// Equivalent to:

IEnumerable<int> source3 = Enumerable.Range(0, 5);

source3.Share(source => source.Concat(source)).WriteLines(); // 0 1 2 3 4

}

The above 2 kinds of Share usage are equivalent. As already discussed, Concat can be desugared as:

public static IEnumerable<TSource> Concat<TSource>(

IEnumerable<TSource>first, IEnumerable<TSource> second)

using (IEnumerator<TSource> iterator1 = first.GetEnumerator())

while (iterator1.MoveNext())

yield return iterator1.Current;

using (IEnumerator<TSource> iterator2 = second.GetEnumerator())

while (iterator2.MoveNext())

yield return iterator2.Current;

}

So that the above 3 Concat calls can be virtually viewed as:

internal static void DesugaredConcatShared()

IEnumerable<int>source1 = Enumerable.Range(0, 5);

IEnumerable<int>Concat1() // source1.Concat(source1)

using (IEnumerator<int> independentIterator1 = source1.GetEnumerator())

while (independentIterator1.MoveNext())

yield return independentIterator1.Current; // Yield 0 1 2 3 4.

using (IEnumerator<int> independentIterator2 = source1.GetEnumerator())

while (independentIterator2.MoveNext())

yield return independentIterator2.Current; // Yield 0 1 2 3 4.

Concat1().WriteLines();

using (IBuffer<int> source2 = Enumerable.Range(0, 5).Share())

IEnumerable<int>Concat2() // source2.Concat(source2)

using (IEnumerator<int> sharedIterator1 = source2.GetEnumerator())

while (sharedIterator1.MoveNext())

yield return sharedIterator1.Current; // Yield 0 1 2 3 4.

using (IEnumerator<int> sharedIterator2 = source2.GetEnumerator())

while (sharedIterator2.MoveNext())

yield return sharedIterator2.Current; // Yield nothing.

Concat2().WriteLines();

IEnumerable<int>source3 = Enumerable.Range(0, 5);

IEnumerable<int>Concat3() // source3.Share(source => source.Concat(source))

using (IBuffer<int> source = source3.Share())

using (IEnumerator<int> sharedIterator1 = source.GetEnumerator())

while (sharedIterator1.MoveNext())

yield return sharedIterator1.Current; // Yield 0 1 2 3 4.

using (IEnumerator<int> sharedIterator2 = source.GetEnumerator())

while (sharedIterator2.MoveNext())

yield return sharedIterator2.Current; // Yield nothing.

Concat3().WriteLines();

}

When Concat is executed, if values are pulled from 2 independent iterators, both iterators yield all source values; if values are pulled from 2 shared iterators. only the first iterator yields all source values, and the second iterator yields nothing. Another example is Zip:

internal static void ZipShared()

IEnumerable<int>source1 = Enumerable.Range(0, 5);

source1.Zip(source1, ValueTuple.Create).WriteLines(); // (0, 0) (1, 1) (2, 2) (3, 3) (4, 4)

using (IBuffer<int> source2 = Enumerable.Range(0, 5).Share())

source2.Zip(source2, ValueTuple.Create).WriteLines(); // (0, 1) (2, 3)

// Equivalent to:

IEnumerable<int> source3 = Enumerable.Range(0, 5);

source3.Share(source => source.Zip(source, ValueTuple.Create)).WriteLines(); // (0, 1) (2, 3).

}

Similarly, the above 3 Zip calls can be virtually viewed as:

internal static void DesugaredZipShared()

IEnumerable<int>source1 = Enumerable.Range(0, 5);

IEnumerable<(int, int)> Zip1()

using (IEnumerator<int> independentIterator1 = source1.GetEnumerator())

using (IEnumerator<int> independentIterator2 = source1.GetEnumerator())

while (independentIterator1.MoveNext() && independentIterator2.MoveNext())

yield return (independentIterator1.Current, independentIterator2.Current);

// Yield (0, 0) (1, 1) (2, 2) (3, 3) (4, 4).

Zip1().WriteLines();

using (IBuffer<int> source2 = Enumerable.Range(0, 5).Share())

IEnumerable<(int, int)> Zip2()

using (IEnumerator<int> sharedIterator1 = source2.GetEnumerator())

using (IEnumerator<int> sharedIterator2 = source2.GetEnumerator())

while (sharedIterator1.MoveNext() && sharedIterator2.MoveNext())

yield return (sharedIterator1.Current, sharedIterator2.Current);

// Yield (0, 1) (2, 3).

Zip2().WriteLines();

IEnumerable<int>source3 = Enumerable.Range(0, 5);

IEnumerable<(int, int)> Zip3()

using (IBuffer<int> source = source3.Share())

using (IEnumerator<int> sharedIterator1 = source.GetEnumerator())

using (IEnumerator<int> sharedIterator2 = source.GetEnumerator())

while (sharedIterator1.MoveNext() && sharedIterator2.MoveNext())

yield return (sharedIterator1.Current, sharedIterator2.Current);

// yields (0, 1) (2, 3).

Zip3().WriteLines();

}

Publish has the same signatures as Share:

public static IBuffer<TSource> Publish<TSource>(this IEnumerable<TSource> source);

public static IEnumerable<TResult> Publish<TSource, TResult>(

this IEnumerable<TSource> source, Func<IEnumerable<TSource>, IEnumerable<TResult>>selector);

It also buffers the values in a different way, so each iterator yields all remainder values:

internal static void Publish()

using (IBuffer<int> publish = Enumerable.Range(0, 5).Publish())

IEnumerator<int>remainderIteratorA = publish.GetEnumerator();

// remainderIteratorA: 0 1 2 3 4. A|B|C

remainderIteratorA.MoveNext(); remainderIteratorA.Current.WriteLine(); // 0| |

remainderIteratorA.MoveNext(); remainderIteratorA.Current.WriteLine(); // 1| |

remainderIteratorA.MoveNext(); remainderIteratorA.Current.WriteLine(); // 2| |

IEnumerator<int> remainderIteratorB = publish.GetEnumerator(); // | |

// remainderIteratorB: 3 4. | |

remainderIteratorB.MoveNext(); remainderIteratorB.Current.WriteLine(); // |3|

remainderIteratorA.MoveNext(); remainderIteratorA.Current.WriteLine(); // 3| |

IEnumerator<int> remainderIteratorC = publish.GetEnumerator(); // | |

// remainderIteratorC: 4. | |

remainderIteratorB.MoveNext(); remainderIteratorB.Current.WriteLine(); // |4|

remainderIteratorA.MoveNext(); remainderIteratorA.Current.WriteLine(); // 4| |

remainderIteratorC.MoveNext(); remainderIteratorC.Current.WriteLine(); // | |4

}

Memoize (not Memorize) simply buffers all values:

public static IBuffer<TSource> Memoize<TSource>(this IEnumerable<TSource> source);

public static IEnumerable<TResult> Memoize<TSource, TResult>(

this IEnumerable<TSource> source, Func<IEnumerable<TSource>, IEnumerable<TResult>>selector);

The term memoize/memoization means buffering the function call result, so that when the same call happens again, the buffered result can be returned. Its multiple iterators work like independent, but each value is only pulled once and is buffered for reuse:

internal static void Memoize()

using (IBuffer<int> memoize = Enumerable.Range(0, 5).Memoize())

IEnumerator<int>bufferIteratorA = memoize.GetEnumerator();

// bufferIteratorA: 0 1 2 3 4. A|B|C

bufferIteratorA.MoveNext(); bufferIteratorA.Current.WriteLine(); // 0| |

bufferIteratorA.MoveNext(); bufferIteratorA.Current.WriteLine(); // 1| |

bufferIteratorA.MoveNext(); bufferIteratorA.Current.WriteLine(); // 2| |

IEnumerator<int> bufferIteratorB = memoize.GetEnumerator(); // | |

// bufferIteratorB: 0 1 2 3 4. | |

bufferIteratorB.MoveNext(); bufferIteratorB.Current.WriteLine(); // |0|

bufferIteratorA.MoveNext(); bufferIteratorA.Current.WriteLine(); // 3| |

IEnumerator<int> bufferIteratorC = memoize.GetEnumerator(); // | |

// bufferIteratorC: 0 1 2 3 4. | |

bufferIteratorB.MoveNext(); bufferIteratorB.Current.WriteLine(); // |1|

bufferIteratorA.MoveNext(); bufferIteratorA.Current.WriteLine(); // 4| |

bufferIteratorC.MoveNext(); bufferIteratorC.Current.WriteLine(); // | |0

bufferIteratorC.MoveNext(); bufferIteratorC.Current.WriteLine(); // | |1

bufferIteratorB.MoveNext(); bufferIteratorB.Current.WriteLine(); // |2|

// ...

}

There 2 more overloads accept a readerCount to specify how many times can the buffered values be reused:

public static IBuffer<TSource> Memoize<TSource>(

this IEnumerable<TSource> source, int readerCount);

public static IEnumerable<TResult> Memoize<TSource, TResult>(

this IEnumerable<TSource> source, int readerCount, Func<IEnumerable<TSource>, IEnumerable<TResult>> selector);

When exceeding the readerCount, an InvalidOperationException is thrown:

internal static void MemoizeWithReaderCount()

using (IBuffer<int> source1 = Enumerable.Range(0, 5).Memoize(2))

int[] reader1 = source1.ToArray(); // First full iteration.

int[] reader2 = source1.ToArray(); // Second full iteration.

int[] reader3 = source1.ToArray(); // Third full iteration: InvalidOperationException.

IEnumerable<int>source2 = Enumerable.Range(0, 5);

source2

.Memoize(

readerCount: 2,

selector: source => source // First full iteration.

.Concat(source) // Second full iteration.

.Concat(source)) // Third full iteration: InvalidOperationException.

.WriteLines();

}

Exception handling

The exception queries address some exception related scenarios for IEnumerable<T>. Throw query just throws the specified exception when executed:

public static IEnumerable<TResult> Throw<TResult>(Exception exception)

throw exception;

yield break; // Deferred execution.

}

The yield break statement at the end is required for deferred execution. Without the yield break statement, the specified exception is thrown immediately when Throw is called. With the yield break statement, a generator is returned when Throw is called, and the specified exception is thrown when trying to pull value from the returned generator for the first time. For example:

internal static void Throw()

IEnumerable<int>@throw = EnumerableEx.Throw<int>(new OperationCanceledException());

IEnumerable<int>query = Enumerable.Range(0, 5).Concat(@throw); // Define query.

try

foreach (int value in query) // Execute query.

value.WriteLine();

catch (OperationCanceledException exception)

exception.WriteLine();

// 0 1 2 3 4 System.OperationCanceledException: The operation was canceled.

}

Catch accepts a source sequence and an exception handler function. When the query is executed, it pulls and yields each value from source sequence. If there is no exception of the specified type thrown during the evaluation, the handler is not called. If any exception of the specified type is thrown, it calls the exception handler with the exception. The handler returns a sequence, whose values are then pulled and yielded. So, Catch’s concept can be virtually viewed as:

// Cannot be compiled.

public static IEnumerable<TSource> CatchWithYield<TSource, TException>(

this IEnumerable<TSource> source, Func<TException, IEnumerable<TSource>> handler)

where TException : Exception

try

foreach (TSource value in source)

yield return value; // Deferred execution.

catch (TException exception)

foreach (TSource value in handler(exception) ?? Empty<TSource>())

yield return value; // Deferred execution.

}

However, C# does not support yield statement inside try-catch statement. The above code cannot be compiled. The solution is to desugar the foreach statement to a while loop for iterator. Then the try-catch statement can go inside the loop, and only contains iterator’s MoveNext and Current calls, and the yield statement can go outside the try-catch statement.

public static IEnumerable<TSource> Catch<TSource, TException>(

this IEnumerable<TSource> source, Func<TException, IEnumerable<TSource>> handler)

where TException : Exception

TException firstException = null;

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (true)

TSource value;

try // Only MoveNext and Current are inside try-catch.

if (iterator.MoveNext())

value = iterator.Current;

else

break; // Stops while loop at the end of iteration.

catch (TException exception)

firstException = exception;

break; // Stops while loop if TException is thrown.

yield return value; // Deferred execution, outside try-catch.

if (firstException != null)

foreach (TSource value in handler(firstException) ?? Empty<TSource>())

yield return value; // Deferred execution.

}

And here is a simple example:

internal static void CatchWithHandler()

IEnumerable<string> @throw = EnumerableEx.Throw<string>(

new OperationCanceledException());

IEnumerable<string>@catch = @throw.Catch<string, OperationCanceledException>(

exception => EnumerableEx.Return($"Handled {exception.GetType().Name}: {exception.Message}"));

@catch.WriteLines(); // Handled OperationCanceledException: The operation was canceled.

}

The other Catch overloads accepts multiple sequences, and outputs a single sequence. The idea is, when executed, it tries to pull and yield values of the first source sequence. if there is no exception, it stops execution; If any exception is thrown, it tries to pull and yield the values of the second source sequence, and so on; When stopping the evaluation, if there is any exception from the evaluation of the last sequence. If yes, it re-throws that exception. The concept is:

// Cannot be compiled.

public static IEnumerable<TSource> CatchWithYield<TSource>(

this IEnumerable<IEnumerable<TSource>> sources)

Exception lastException = null;

foreach (IEnumerable<TSource> source in sources)

lastException = null;

try

foreach (TSource value in source)

yield return value; // Deferred execution.

break; // Stops if no exception from current sequence.

catch (Exception exception)

lastException = exception;

// Continue with next sequence if there is exception.

if (lastException != null)

throw lastException;

}

Again, the above code cannot be compiled because yield statement cannot be used with try-catch statement. So previous desugared while-try-catch-yield pattern can be used:

public static IEnumerable<TSource> Catch<TSource>(

this IEnumerable<IEnumerable<TSource>> sources)

Exception lastException = null;

foreach (IEnumerable<TSource> source in sources)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (true)

lastException = null;

TSource value;

try // Only MoveNext and Current are inside try-catch.

if (iterator.MoveNext())

value = iterator.Current;

else

break; // Stops while loop at the end of iteration.

catch (Exception exception)

lastException = exception;

break; // Stops while loop if TException is thrown.

yield return value; // Deferred execution, outside try-catch.

if (lastException == null)

break; // If no exception, stops pulling the next source; otherwise, continue.

if (lastException != null)

throw lastException;

public static IEnumerable<TSource> Catch<TSource>(

params IEnumerable<TSource>[] sources) => sources.Catch();

public static IEnumerable<TSource> Catch<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second) =>

new IEnumerable<TSource>[] { first, second }.Catch();

For example:

internal static void Catch()

IEnumerable<int>scanWithException = Enumerable.Repeat(0, 5).Scan((a, b) => a / b); // Divide by 0.

IEnumerable<int> range = Enumerable.Range(0, 5);

IEnumerable<int>castWithException = new object[] { 5, "a" }.Cast<int>();

IEnumerable<IEnumerable<int>> source1 = new IEnumerable<int>[]

scanWithException, // Executed, with DivideByZeroException.

range, // Executed, without exception.

castWithException // Not executed.

};

source1.Catch().WriteLines(); // 0 1 2 3 4

IEnumerable<IEnumerable<int>> source2 = new IEnumerable<int>[]

scanWithException, // Executed, with DivideByZeroException.

castWithException // Executed, with InvalidCastException.

};

try

source2.Catch().WriteLines(); // 5

catch (InvalidCastException exception)

exception.WriteLine(); // System.InvalidCastException: Specified cast is not valid.

}

Besides Throw and Catch, there is also Finally query. Finally is very intuitive:

public static IEnumerable<TSource> Finally<TSource>(this IEnumerable<TSource> source, Action finalAction)

try

foreach (TSource value in source)

yield return value; // Deferred execution.

finally

finalAction();

}

The above code can be compiled because yield statement is allowed in the try block of try-finally statement.

OnErrorResumeNext is similar to Concat, but it ignores any exception when evaluating values from each sequence. The idea is:

// Cannot be compiled.

internal static IEnumerable<TSource> OnErrorResumeNextWithYield<TSource>(

this IEnumerable<IEnumerable<TSource>> sources)

foreach (IEnumerable<TSource> source in sources)

try

foreach (TSource value in source)

yield return value; // Deferred execution.

catch { }

}

Once again, this can be implemented with the desugared while-try-catch-yield pattern:

public static IEnumerable<TSource> OnErrorResumeNext<TSource>(

this IEnumerable<IEnumerable<TSource>> sources)

foreach (IEnumerable<TSource> source in sources)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (true)

TSource value = default;

try

if (!iterator.MoveNext())

break;

value = iterator.Current;

catch

break;

yield return value; // Deferred execution.

public static IEnumerable<TSource> OnErrorResumeNext<TSource>(

params IEnumerable<TSource>[] sources) => sources.OnErrorResumeNext();

public static IEnumerable<TSource> OnErrorResumeNext<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second) =>

new IEnumerable<TSource>[] { first, second }.OnErrorResumeNext();

Retry query tries to yield the source values. If there is an exception thrown, it retries to yield the values again from the beginning of the source sequence. Its implementation is equivalent to:

public static IEnumerable<TSource> Retry<TSource>(

this IEnumerable<TSource> source, int? retryCount = null) =>

Return(source).Repeat(retryCount).Catch();

If retryCount is not provided, it retries forever.

Control flow

The If/Case/Using/While/DoWhile/Generate/For queries implements the control flows as fluent LINQ query. If represents the if-else statement. Its implementation is equivalent to:

public static IEnumerable<TResult> If<TResult>(

Func<bool>condition, IEnumerable<TResult> thenSource, IEnumerable<TResult> elseSource = null) =>

Defer(() => condition() ? thenSource : elseSource ?? Enumerable.Empty<TResult>());

Case represents the switch-case statement. It accepts a selector function as the key factory, and a dictionary of key-sequence pairs, where each key represents a case label of the switch statement. When Case query is executed, the selector function is called to get a key. If the dictionary contains that key, then the matching sequence is the query output; otherwise, a default sequence is the query output:

public static IEnumerable<TResult> Case<TValue, TResult>(

Func<TValue>selector,

IDictionary<TValue, IEnumerable<TResult>>sources,

IEnumerable<TResult>defaultSource = null) =>

Defer(() => sources.TryGetValue(selector(), out IEnumerable<TResult>result)

? result

: (defaultSource ?? Enumerable.Empty<TResult>()));

Using represents the using statement:

public static IEnumerable<TSource> Using<TSource, TResource>(

Func<TResource>resourceFactory, Func<TResource, IEnumerable<TSource>> enumerableFactory)

where TResource : IDisposable

using (TResource resource = resourceFactory())

foreach (TSource value in enumerableFactory(resource))

yield return value; // Deferred execution.

}

While represents the while loop:

public static IEnumerable<TResult> While<TResult>(Func<bool> condition, IEnumerable<TResult> source)

while (condition())

foreach (TResult value in source)

yield return value; // Deferred execution.

}

DoWhile represents the do-while loop:

public static IEnumerable<TResult> DoWhile<TResult>(

this IEnumerable<TResult> source, Func<bool> condition) =>

source.Concat(While(condition, source));

Generate represents the for loop:

public static IEnumerable<TResult> Generate<TState, TResult>(

TState initialState,

Func<TState, bool> condition,

Func<TState, TState> iterate,

Func<TState, TResult> resultSelector)

for (TState state = initialState; condition(state); state = iterate(state))

yield return resultSelector(state); // Deferred execution.

}

For also works the same as SelectMany. Its implementation is equivalent to:

public static IEnumerable<TResult> For<TSource, TResult>(

IEnumerable<TSource>source, Func<TSource, IEnumerable<TResult>>resultSelector) =>

source.SelectMany(resultSelector);

It can be viewed as foreach statement – for each value in the source, call the resultSelector function and yields all results in the function’s output sequence. I am not sure why the 2 above queries are named as Generate and For.

Iteration

Do does not transform the data in any way. It simply pulls source values just like Hide. It also accepts 3 callback functions, onNext, onError, and onCompleted. When each source value is pulled, onNext is called with the value. When exception is thrown for pulling source value, onError is called with the exception. After all source values are pulled successfully without exception, onCompleted is called. Its idea is:

public static IEnumerable<TSource> Do<TSource>(

this IEnumerable<TSource> source,

Action<TSource> onNext, Action<Exception> onError = null, Action onCompleted = null)

try

foreach (TSource value in source)

onNext(value);

yield return value;

catch (Exception exception)

onError?.Invoke(exception);

throw;

onCompleted?.Invoke();

}

Once again, the yield statement does not work with try-catch statement. The above idea can be implemented with the desugared while-try-catch-yield pattern:

public static IEnumerable<TSource> Do<TSource>(

this IEnumerable<TSource> source,

Action<TSource>onNext, Action<Exception>onError = null, Action onCompleted = null)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (true)

TSource value;

try

if (!iterator.MoveNext())

break;

value = iterator.Current;

catch (Exception exception)

onError?.Invoke(exception);

throw;

onNext(value);

yield return value; // Deferred execution, outside try-catch.

onCompleted?.Invoke();

}

Do is very useful for logging and tracing LINQ queries, for example:

internal static void Do()

Enumerable

.Range(-5, 10).Do(

onNext: value => $"{nameof(Enumerable.Range)} yields {value}.".WriteLine(),

onCompleted: () => $"{nameof(Enumerable.Range)} completes.".WriteLine())

.Where(value => value > 0).Do(

onNext: value => $"{nameof(Enumerable.Where)} yields {value}.".WriteLine(),

onCompleted: () => $"{nameof(Enumerable.Where)} completes.".WriteLine())

.TakeLast(2).Do(

onNext: value => $"{nameof(EnumerableEx.TakeLast)} yields {value}.".WriteLine(),

onCompleted: () => $"{nameof(EnumerableEx.TakeLast)} completes.".WriteLine())

.WriteLines(value => $"Composited query yields result {value}.");

// Range yields -5.

// Range yields -4.

// Range yields -3.

// Range yields -2.

// Range yields -1.

// Range yields 0.

// Range yields 1.

// Where yields 1.

// Range yields 2.

// Where yields 2.

// Range yields 3.

// Where yields 3.

// Range yields 4.

// Where yields 4.

// Range completes.

// Where completes.

// TakeLast yields 3.

// Composited query yields result 3.

// TakeLast yields 4.

// Composited query yields result 4.

// TakeLast completes.

}

Since System.IObserver<T> is the composition of above onNext, onError, onCompleted functions:

namespace System

public interface IObserver<in T>

void OnCompleted();

void OnError(Exception error);

void OnNext(T value);

}

Do also has an overload accepting an observer:

public static IEnumerable<TSource> Do<TSource>(this IEnumerable<TSource> source, IObserver<TSource> observer) =>

Do(source, observer.OnNext, observer.OnError, observer.OnCompleted);

Value queries

Ix provides a few queries for finding the extremum as well as empty test:

Aggregation

The additional overloads of Max/Min accept a comparer function, and return the first maximum/minimum value:

public static TSource Max<TSource>(

this IEnumerable<TSource> source, IComparer<TSource> comparer);

public static TSource Min<TSource>(

this IEnumerable<TSource> source, IComparer<TSource> comparer);

As fore mentioned, to use the standard Max/Min with a source sequence, exception is thrown if the source type does not implement IComparable or IComparable<T>, which is a problem when the source type cannot be modified to add IComparable or IComparable<T> implementation:

internal static void MaxMinGeneric()

Character maxCharacter = Characters().Max().WriteLine();

Character minCharacter = Characters().Min().WriteLine();

}

The overloads with comparer does not have such requirement:

internal static void MaxMin()

Character maxCharacter = Characters()

.Max(Comparer<Character>.Create((character1, character2) => string.Compare(

character1.Name, character2.Name, StringComparison.OrdinalIgnoreCase)));

Character minCharacter = Characters()

.Max(Comparer<Character>.Create((character1, character2) => string.Compare(

character1.Name, character2.Name, StringComparison.OrdinalIgnoreCase)));

}

MaxBy/MinBy accept key selector and key comparer functions, and their output is a list of all maximum/minimum values:

public static IList<TSource> MaxBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IList<TSource> MaxBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector, IComparer<TKey> comparer);

public static IList<TSource> MinBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IList<TSource> MinBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector, IComparer<TKey> comparer);

For example:

internal static void MaxByMinBy()

IList<Character>maxCharacters = Characters()

.MaxBy(character => character.Name, StringComparer.OrdinalIgnoreCase);

IList<Character>minCharacters = Characters()

.MinBy(character => character.Name, StringComparer.OrdinalIgnoreCase);

}

The previous example of finding the maximum types in core library becomes easy with MaxBy:

internal static void MaxBy()

CoreLibrary.ExportedTypes

.Select(type => (Type: type, MemberCount: type.GetDeclaredMembers().Length))

.MaxBy(typeAndMemberCount => typeAndMemberCount.MemberCount)

.WriteLines(max => $"{max.Type.FullName}:{max.MemberCount}"); // System.Convert:311

}

Quantifiers

There is an IsEmpty query for convenience. It is just the opposite of Any:

public static bool IsEmpty<TSource>(this IEnumerable<TSource> source) => !source.Any();

Void queries

Ix provides a ForEach query to iterate the source sequence, which is similar to List<T>.ForEach method.

Iteration

ForEach represents the foreach loop, with a non-indexed overload and an indexed overload, which can be fluently used at the end of LINQ query. This is probably the handiest query in LINQ programming, because it executes the LINQ query and process the query results:

public static void ForEach<TSource>(

this IEnumerable<TSource> source, Action<TSource> onNext)

foreach (TSource value in source)

onNext(value);

public static void ForEach<TSource>(

this IEnumerable<TSource> source, Action<TSource, int>onNext)

int index = 0;

foreach (TSource value in source)

onNext(value, index);

index = checked(index + 1);

}

There was an issue with the indexed ForEach – the index increment was not checked. The issue was uncovered when writing this book and has been fixed.

Summary

This chapter discusses the additional LINQ to Objects queries provided by Microsoft through Ix, including sequence queries for generation, filtering, mapping, concatenation, set, partitioning, conversion, buffering, exception, control flow, iteration, value queries for aggregation, quantifiers, and the handiest ForEach to execute LINQ query.

LINQ to Objects in Depth (5) Query Methods Implementation

Fri, 05 Jul 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to Objects in Depth series]

Understanding of internal implementation of LINQ to Objects queries is the ultimate way to master them and use them accurately and effectively, and is also helpful for defining custom query methods, which is discussed later in the custom queries chapter. Just like the usage discussion part, here query methods are still categorized by output type, but in a different order:

Collection queries: output a new collection (immediate execution):

o Conversion: ToArray, ToList, ToDictionary, ToLookup

Sequence queries: output a new IEnumerable<T> sequence (deferred execution, underlined are eager evaluation):

o Conversion: Cast, AsEnumerable

o Generation: Empty, Range, Repeat, DefaultIfEmpty

o Filtering (restriction): Where, OfType

o Mapping (projection): Select, SelectMany

o Grouping: GroupBy*

o Join: SelectMany, Join*, GroupJoin*

o Concatenation: Concat

o Set: Distinct, Union, Intersect*, Except*

o Convolution: Zip

o Partitioning: Take, Skip, TakeWhile, SkipWhile

o Ordering: OrderBy*, ThenBy*, OrderByDescending*, ThenByDescending*, Reverse*

Value queries: output a single value (immediate execution):

o Element: First, FirstOrDefault, Last, LastOrDefault, ElementAt, ElementAtOrDefault, Single, SingleOrDefault

o Aggregation: Aggregate, Count, LongCount, Min, Max, Sum, Average

o Quantifier: All, Any, Contains

o Equality: SequenceEqual

The LINQ to Objects queries are functional as APIs, but their implementation is imperative. Again, the collection queries and value queries implement immediate execution, and the sequence queries implements deferred execution. For the sequential queries, the ones marked with * implements eager evaluation, and the other ones without mark implements lazy evaluation.

Argument check and deferred execution

As fore mentioned, all sequence queries implement deferred execution, mostly by following iterator pattern, which can be easily implemented with yield statement. For a generator function with the yield syntactic sugar, its arguments check cannot be directly added to the function body, because the execution of all code in the body is deferred. For example, assuming arguments check is added to Select query as the following:

internal static IEnumerable<TResult> SelectWithDeferredCheck<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector)

if (source == null)

throw new ArgumentNullException(nameof(source));

if (selector == null)

throw new ArgumentNullException(nameof(selector));

foreach (TSource value in source)

yield return selector(value); // Deferred execution.

}

Its arguments are expected to be checked immediately when it is called. However, the check is deferred. When it is called, it only constructs the following generator:

internal static IEnumerable<TResult> CompiledSelectWithDeferredCheck<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector)

IEnumerator<TSource> sourceIterator = null;

return new Generator<TResult>(

start: () =>

if (source == null)

throw new ArgumentNullException(nameof(source));

if (selector == null)

throw new ArgumentNullException(nameof(selector));

sourceIterator = source.GetEnumerator();

},

moveNext: () => sourceIterator.MoveNext(),

getCurrent: () => selector(sourceIterator.Current),

dispose: () => sourceIterator?.Dispose());

}

The argument check is deferred to execute when starting to pull the results from the output sequence. The easiest solution is simply isolating yield statement along with its iteration control flow, which is compiled to deferred execution, with a separate method or a local function:

internal static IEnumerable<TResult> SelectWithCheck<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector)

if (source == null) // Immediate execution.

throw new ArgumentNullException(nameof(source));

if (selector == null) // Immediate execution.

throw new ArgumentNullException(nameof(selector));

IEnumerable<TResult> SelectGenerator()

foreach (TSource value in source)

yield return selector(value); // Deferred execution.

return SelectGenerator(); // Immediate execution.

}

As a result, the above outer function is no longer a generator function. When it is called, it immediately checks the arguments, then immediately calls the local function to construct a generator as output. As mentioned in the introduction chapter, this tutorial omits argument null checks and only demonstrate argument checks for other purposes .

Collection queries

The collection queries are discussed first because they are relatively straightforward, and they are used to implement sequence queries:

Conversion

ToArray is implemented by pulling all values from source sequence and store them to a new array. To create an array, its length has to be provided. However, the count of values in source is unknown when starting to pull the values. The easiest implementation is to create an empty array, when each value is pulled from source sequence, increase the array size by 1 to store that value:

internal static partial class EnumerableExtensions

public static TSource[] ToArray<TSource>(this IEnumerable<TSource> source)

TSource[] array = new TSource[0];

foreach (TSource value in source)

Array.Resize(ref array, array.Length + 1);

array[array.Length - 1] = value;

return array;

}

Actually, Microsoft’s built-in implementation has some performance improvement. First, if the source sequence implements ICollection<T>, then it already has a CopyTo method to store its values to an array:

namespace System.Collections.Generic

public interface ICollection<T> : IEnumerable<T>, IEnumerable

int Count { get; }

bool IsReadOnly { get; }

void Add(T item);

void Clear();

bool Contains(T item);

void CopyTo(T[] array, int arrayIndex);

bool Remove(T item);

}

Also, the array does not need to be resized for each value. The built-in implementation has the arrays grow in the same way of List<T>. First, an initial length is used to construct the array; when pulling values from source and storing to array, if array gets full, then double its length; After all values are pulled, the array resized to the actual length. The following is an equivalent implementation, optimized for readability:

public static TSource[] ToArray<TSource>(this IEnumerable<TSource> source)

if (source is ICollection<TSource> genericCollection)

int length = genericCollection.Count;

if (length > 0)

TSource[] array = new TSource[length];

genericCollection.CopyTo(array, 0);

return array;

else

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (iterator.MoveNext())

const int InitialLength = 4; // Initial array length.

const int MaxLength = 0x7FEFFFFF; // Max array length: Array.MaxArrayLength.

TSource[] array = new TSource[InitialLength];

array[0] = iterator.Current;

int usedLength = 1;

while (iterator.MoveNext())

if (usedLength == array.Length)

int increaseToLength = usedLength * 2; // Array is full, double its size.

if ((uint)increaseToLength> MaxLength)

increaseToLength = usedLength> = MaxLength ? usedLength + 1 : MaxLength;

Array.Resize(ref array, increaseToLength);

array[usedLength++] = iterator.Current;

Array.Resize(ref array, usedLength); // Consolidate array to its actual size.

return array;

return Array.Empty<TSource>();

}

ToList is much easier to implement, because List<T> has a constructor accepting an IEnumerable<T> source:

public static List<TSource> ToList<TSource>(this IEnumerable<TSource> source) =>

new List<TSource>(source);

ToDictionary is also easy, because Dictionary<TKey, TValue> has an Add method:

public static Dictionary<TKey, TSource> ToDictionary<TSource, TKey>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IEqualityComparer<TKey>comparer = null) =>

source.ToDictionary(keySelector, value => value, comparer);

public static Dictionary<TKey, TElement> ToDictionary<TSource, TKey, TElement>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

IEqualityComparer<TKey>comparer = null)

Dictionary<TKey, TElement> dictionary = new Dictionary<TKey, TElement>(comparer);

foreach (TSource value in source)

dictionary.Add(keySelector(value), elementSelector(value));

return dictionary;

}

As previously discussed, a lookup is a dictionary of key and sequence pairs, and each key and sequence pair are just a group represented by IGrouping<TKey, TElement>, which can be implemented as:

public class Grouping<TKey, TElement> : IGrouping<TKey, TElement>

private readonly List<TElement> values = new List<TElement>();

public Grouping(TKey key) => this.Key = key;

public TKey Key { get; }

public IEnumerator<TElement> GetEnumerator() => this.values.GetEnumerator();

IEnumerator IEnumerable.GetEnumerator() => this.GetEnumerator();

internal void Add(TElement value) => this.values.Add(value);

}

.NET provides a public lookup type, but there is no public API to instantiate it, except the ToLookup query itself. For demonstration purpose, based on the previous discussion of dictionary and lookup, a custom lookup can be quickly implemented with dictionary, where each dictionary value is a group, and each dictionary key is the hash code of group key:

public partial class Lookup<TKey, TElement> : ILookup<TKey, TElement>

private readonly Dictionary<int, Grouping<TKey, TElement>> groups =

new Dictionary<int, Grouping<TKey, TElement>>();

private readonly IEqualityComparer<TKey> equalityComparer;

public Lookup(IEqualityComparer<TKey> equalityComparer = null) =>

this.equalityComparer = equalityComparer ?? EqualityComparer<TKey>.Default;

public int Count => this.groups.Count;

private int GetHashCode(TKey key) => key == null

? -1

: this.equalityComparer.GetHashCode(key) & int.MaxValue;

// int.MaxValue is 0b_01111111_11111111_11111111_11111111. So the hash code of non-null key is always > -1.

public bool Contains(TKey key) => this.groups.ContainsKey(this.GetHashCode(key));

public IEnumerable<TElement> this[TKey key] =>

this.groups.TryGetValue(this.GetHashCode(key), out Grouping<TKey, TElement> group)

? (IEnumerable<TElement>)group

: Array.Empty<TElement>();

public IEnumerator<IGrouping<TKey, TElement>> GetEnumerator() => this.groups.Values.GetEnumerator();

IEnumerator IEnumerable.GetEnumerator() => this.GetEnumerator();

}

The built-in API object.GetHashCode is not directly used to get each group key’s hash code, because it does not handle null value very well in some cases. System.Nullable<T>.GetHashCode is such an example. ((int?)0).GetHashCode() and ((int?)null).GetHashCode() both return 0. So, the above GetHashCode method reserves -1 for null. And any non-null value’s hash code is converted to a positive int by a bitwise and operation with int.MaxValue (0b_01111111_11111111_11111111_11111111). Also notice the indexer getter outputs an empty sequence when the specified key does not exist. Similar to Grouping<TKey, TElement>.Add, the following Lookup<TKey, TElement>.AddRange is defined to add data:

public partial class Lookup<TKey, TElement>

public Lookup<TKey, TElement> AddRange<TSource>(

IEnumerable<TSource>source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

bool skipNullKey = false)

foreach (TSource value in source)

TKey key = keySelector(value);

if (key == null &&skipNullKey)

continue;

int hashCode = this.GetHashCode(key);

if (this.groups.TryGetValue(hashCode, out Grouping<TKey, TElement> group))

group.Add(elementSelector(value));

else

this.groups.Add(hashCode, new Grouping<TKey, TElement>(key) { elementSelector(value) });

return this;

}

Now, ToLookup can be implemented by creating a lookup and adding all data:

public static ILookup<TKey, TElement> ToLookup<TSource, TKey, TElement>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

IEqualityComparer<TKey>comparer = null) =>

new Lookup<TKey, TElement>(comparer).AddRange(source, keySelector, elementSelector);

public static ILookup<TKey, TSource> ToLookup<TSource, TKey>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IEqualityComparer<TKey>comparer = null) =>

source.ToLookup(keySelector, value => value, comparer);

Sequence queries

All sequence queries implement deferred execution. Most of them use follows the iterator pattern with a generator. For these queries, some are implemented with the yield syntactic sugar, and some are implemented with custom defined generator for performance improvement. Here, to make it intuitive and readable, all those queries’ implementation is demonstrated with yield.

Conversion

AsEnumerable does nothing:

public static IEnumerable<TSource> AsEnumerable<TSource>(this IEnumerable<TSource> source) =>

source; // Deferred execution.

With an IEnumerable<T> output, it just makes the source’s non-sequence methods (if any) not available. It also implements deferred execution, because calling AsEnumerable does not pull any value from source sequence.

Cast is very easy to implement with the generator syntactic sugar. Just yield each cast value:

public static IEnumerable<TResult> Cast<TResult>(this IEnumerable source)

foreach (object value in source)

yield return (TResult)value; // Deferred execution.

}

The built-in implementation does a little optimization. If the source is already a generic sequence of the specified result type, it can be directly returned. Logically it should be something like:

// Cannot be compiled.

public static IEnumerable<TResult> Cast<TResult>(this IEnumerable source)

if (source is IEnumerable<TResult> genericSource)

return genericSource;

foreach (object value in source)

yield return (TResult)value; // Deferred execution.

}

The above code cannot be compiled. The yield statement indicates the entire function body should be compiled to a generator, so the return statement cannot be used with yield statement. Similar to argument check, the solution is to isolate the iteration control flow with yield statement into another method or local function:

public static IEnumerable<TResult> Cast<TResult>(this IEnumerable source)

IEnumerable<TResult>CastGenerator()

foreach (object value in source)

yield return (TResult)value; // Deferred execution.

return source is IEnumerable<TResult> genericSource

? genericSource

: CastGenerator();

}

Cast also implements deferred execution. When it is called, its output is either the source sequence itself or a generator, without pulling values from source or execute the casting.

Generation

Empty can be implemented with a single yield break statement, which is compiled to an empty generator:

public static IEnumerable<TResult> EmptyGenerator<TResult>()

yield break;

}

The built-in implementation actually uses an empty array, which has better performance than constructing a generator:

public static IEnumerable<TResult> Empty<TResult>() => Array.Empty<TResult>();

Range can be simply implemented with a loop:

public static IEnumerable<int> Range(int start, int count)

if (count < 0 || (((long)start) + count - 1L) > int.MaxValue)

throw new ArgumentOutOfRangeException(nameof(count));

IEnumerable<int>RangeGenerator()

int end = start + count;

for (int value = start; value != end; value++)

yield return value; // Deferred execution.

return RangeGenerator();

}

And Repeat has been discussed. The built-in implementation also checks the count argument, and uses an empty array when count is 0:

public static IEnumerable<TResult> Repeat<TResult>(TResult element, int count)

if (count< 0)

throw new ArgumentOutOfRangeException(nameof(count));

if (count == 0)

return Empty<TResult>();

IEnumerable<TResult> RepeatGenerator()

for (int index = 0; index < count; index++)

yield return element; // Deferred execution.

return RepeatGenerator();

}

DefaultIfEmpty can be implemented with a desugared foreach loop to detect and iterate the source sequence:

public static IEnumerable<TSource> DefaultIfEmpty<TSource>(

this IEnumerable<TSource> source, TSource defaultValue = default)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (iterator.MoveNext())

// source is not empty.

do

yield return iterator.Current; // Deferred execution.

while (iterator.MoveNext());

else

// source is empty.

yield return defaultValue; // Deferred execution.

}

The first MoveNext call detects if the source sequence is empty. If so, just yield the default value, otherwise yield all values in the source sequence.

Filtering

Where is already discussed. The following are the non-indexed overload and index overload equivalent to the built-in implementation:

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source,

Func<TSource, bool> predicate)

foreach (TSource value in source)

if (predicate(value))

yield return value; // Deferred execution.

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source, Func<TSource, int, bool> predicate)

int index = -1;

foreach (TSource value in source)

index = checked(index + 1);

if (predicate(value, index))

yield return value; // Deferred execution.

}

Similarly, OfType has a type test to filter the source:

public static IEnumerable<TResult> OfType<TResult>(this IEnumerable source)

foreach (object value in source)

if (value is TResult)

yield return (TResult)value; // Deferred execution.

}

Mapping

Select has also been discussed. The following are the non-indexed overload and index overload equivalent to the built-in implementation:

public static IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult>selector)

foreach (TSource value in source)

yield return selector(value); // Deferred execution.

public static IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, int, TResult> selector)

int index = -1;

foreach (TSource value in source)

index = checked(index + 1);

yield return selector(value, index); // Deferred execution.

}

The implementation of SelectMany is also straightforward:

public static IEnumerable<TResult> SelectMany<TSource, TResult>(

this IEnumerable<TSource> source,

Func<TSource, IEnumerable<TResult>>selector)

foreach (TSource value in source)

foreach (TResult result in selector(value))

yield return result; // Deferred execution.

}

Above code intuitively shows its capacity to flatten a hierarchical 2-level-sequence to a flat 1-level-sequence. The following is the overload with collection selector and result selector:

public static IEnumerable<TResult> SelectMany<TSource, TCollection, TResult>(

this IEnumerable<TSource> source,

Func<TSource, IEnumerable<TCollection>> collectionSelector,

Func<TSource, TCollection, TResult>resultSelector)

foreach (TSource sourceValue in source)

foreach (TCollection collectionValue in collectionSelector(sourceValue))

yield return resultSelector(sourceValue, collectionValue); // Deferred execution.

}

And the following are the indexed overloads:

public static IEnumerable<TResult> SelectMany<TSource, TResult>(

this IEnumerable<TSource> source,

Func<TSource, int, IEnumerable<TResult>> selector)

int index = -1;

foreach (TSource value in source)

index = checked(index + 1);

foreach (TResult result in selector(value, index))

yield return result; // Deferred execution.

public static IEnumerable<TResult> SelectMany<TSource, TCollection, TResult>(

this IEnumerable<TSource> source,

Func<TSource, int, IEnumerable<TCollection>> collectionSelector,

Func<TSource, TCollection, TResult>resultSelector)

int index = -1;

foreach (TSource sourceValue in source)

index = checked(index + 1);

foreach (TCollection collectionValue in collectionSelector(sourceValue, index))

yield return resultSelector(sourceValue, collectionValue); // Deferred execution.

}

Grouping

GroupBy’s work is similar to ToLookup, but with deferred execution. To implement deferred execution with ToLookup, the easiest way is to involve yield statement:

public static IEnumerable<IGrouping<TKey, TSource>> GroupBy<TSource, TKey>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IEqualityComparer<TKey>comparer = null)

ILookup<TKey, TSource> lookup = source.ToLookup(keySelector, comparer); // Eager evaluation.

foreach (IGrouping<TKey, TSource> group in lookup)

yield return group; // Deferred execution.

}

When starting to pull the first value from the returned generator, ToLookup is called to evaluate all source values and group them, so that the first group can be yielded. Apparently GroupBy implements eager evaluation. The overloads with element selector and resultSelector can all be implemented in the same pattern:

public static IEnumerable<IGrouping<TKey, TElement>> GroupBy<TSource, TKey, TElement>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

IEqualityComparer<TKey>comparer = null)

ILookup<TKey, TElement> lookup = source.ToLookup(keySelector, elementSelector, comparer); // Eager evaluation.

foreach (IGrouping<TKey, TElement> group in lookup)

yield return group; // Deferred execution.

public static IEnumerable<TResult> GroupBy<TSource, TKey, TResult>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TKey, IEnumerable<TSource>, TResult> resultSelector,

IEqualityComparer<TKey>comparer = null)

ILookup<TKey, TSource> lookup = source.ToLookup(keySelector, comparer); // Eager evaluation.

foreach (IGrouping<TKey, TSource> group in lookup)

yield return resultSelector(group.Key, group); // Deferred execution.

public static IEnumerable<TResult> GroupBy<TSource, TKey, TElement, TResult>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

Func<TKey, IEnumerable<TElement>, TResult> resultSelector,

IEqualityComparer<TKey>comparer = null)

ILookup<TKey, TElement> lookup = source.ToLookup(keySelector, elementSelector, comparer); // Eager evaluation.

foreach (IGrouping<TKey, TElement> group in lookup)

yield return resultSelector(group.Key, group); // Deferred execution.

}

Join

Similar to GroupBy, GroupJoin for outer join can be simply implemented with the same pattern of ToLookup and yield:

public static IEnumerable<TResult> GroupJoinWithLookup<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, IEnumerable<TInner>, TResult> resultSelector,

IEqualityComparer<TKey>comparer = null)

ILookup<TKey, TInner> innerLookup = inner.ToLookup(innerKeySelector, comparer); // Eager evaluation.

foreach (TOuter outerValue in outer)

yield return resultSelector(outerValue, innerLookup[outerKeySelector(outerValue)]); // Deferred execution.

}

When trying to pull the first value from the returned generator, the inner values are grouped by the keys, and stored in the inner lookup. Then, for each outer value, query the inner lookup by key. Remember when a lookup is queried with a key, it always outputs a sequence, even when the key does not exist, it returns an empty sequence. So that in GroupJoin, each outer value is always paired with a group of inner values. The above implementation is straightforward, but the inner source is always pulled, even when the outer source is empty. This can be avoided by a little optimization:

public static IEnumerable<TResult> GroupJoin<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner> inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, IEnumerable<TInner>, TResult> resultSelector,

IEqualityComparer<TKey>comparer = null)

using (IEnumerator<TOuter> outerIterator = outer.GetEnumerator())

if (outerIterator.MoveNext())

Lookup<TKey, TInner> innerLookup = new Lookup<TKey, TInner>(comparer).AddRange(

inner, innerKeySelector, innerValue => innerValue, skipNullKey: true); // Eager evaluation.

do

TOuter outerValue = outerIterator.Current;

yield return resultSelector(outerValue, innerLookup[outerKeySelector(outerValue)]); // Deferred execution.

while (outerIterator.MoveNext());

}

Similar to DefaultIfEmpty, the first MoveNext call detects if the outer source is empty. Only if not, the inner values are pulled and converted to a lookup.

Join for inner join can also be implemented with the similar pattern:

public static IEnumerable<TResult> JoinWithToLookup<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, TInner, TResult>resultSelector,

IEqualityComparer<TKey>comparer = null)

ILookup<TKey, TInner> innerLookup = inner.ToLookup(innerKeySelector, comparer); // Eager evaluation.

foreach (TOuter outerValue in outer)

TKey key = outerKeySelector(outerValue);

if (innerLookup.Contains(key))

foreach (TInner innerValue in innerLookup[key])

yield return resultSelector(outerValue, innerValue); // Deferred execution.

}

It calls the ILookup<TKey, TElement>.Contains filter, because in inner join each outer value has to be paired with a matching inner value. Again, the above implementation can be optimized, so that the inner values are not pulled and converted to lookup when the outer source is empty:

public static IEnumerable<TResult> Join<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, TInner, TResult>resultSelector,

IEqualityComparer<TKey>comparer = null)

using (IEnumerator<TOuter> outerIterator = outer.GetEnumerator())

if (outerIterator.MoveNext())

Lookup<TKey, TInner> innerLookup = new Lookup<TKey, TInner>(comparer).AddRange(

inner, innerKeySelector, innerValue => innerValue, skipNullKey: true); // Eager evaluation.

if (innerLookup.Count> 0)

do

TOuter outerValue = outerIterator.Current;

TKey key = outerKeySelector(outerValue);

if (innerLookup.Contains(key))

foreach (TInner innerValue in innerLookup[key])

yield return resultSelector(outerValue, innerValue); // Deferred execution.

while (outerIterator.MoveNext());

}

Concatenation

Concat’s implementation is equivalent to yield values from the first source sequence, then yield values from the from the second:

public static IEnumerable<TSource> Concat<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second)

foreach (TSource value in first)

yield return value; // Deferred execution.

foreach (TSource value in second)

yield return value; // Deferred execution.

}

Append and Prepend’s implementation is equivalent to the following, with similar pattern:

public static IEnumerable<TSource> Append<TSource>(this IEnumerable<TSource> source, TSource element)

foreach (TSource value in source)

yield return value;

yield return element;

public static IEnumerable<TSource> Prepend<TSource>(this IEnumerable<TSource> source, TSource element)

yield return element;

foreach (TSource value in source)

yield return value;

}

Set

All the set queries need to remove duplicate values in the result sequence. So, the following hash set is defined to represent a collection of distinct values. The duplication of values can be identified by their hash codes, so a dictionary can be used to store distinct hash code and value pairs:

public partial class HashSet<T> : IEnumerable<T>

private readonly IEqualityComparer<T> equalityComparer;

private readonly Dictionary<int, T>dictionary = new Dictionary<int, T>();

public HashSet(IEqualityComparer<T> equalityComparer = null) =>

this.equalityComparer = equalityComparer ?? EqualityComparer<T>.Default;

public IEnumerator<T> GetEnumerator() => this.dictionary.Values.GetEnumerator();

IEnumerator IEnumerable.GetEnumerator() => this.GetEnumerator();

}

Then, the following Add and AddRange methods can be defined:

public partial class HashSet<T>

private int GetHashCode(T value) => value == null

? -1

: this.equalityComparer.GetHashCode(value) & int.MaxValue;

// int.MaxValue is 0b_01111111_11111111_11111111_11111111, so the result of & is always > -1.

public bool Add(T value)

int hashCode = this.GetHashCode(value);

if (this.dictionary.ContainsKey(hashCode))

return false;

this.dictionary.Add(hashCode, value);

return true;

public HashSet<T> AddRange(IEnumerable<T> values)

foreach (T value in values)

this.Add(value);

return this;

}

When Add is called with a specified value, if there is already a duplicate hash code in the internal dictionary, the specified value is not stored in the dictionary and false is returned; otherwise, the specified value and its hash code are added to the internal dictionary, and true is returned. With above hash set, it is very easy to implement Distinct.

public static IEnumerable<TSource> Distinct<TSource>(

this IEnumerable<TSource> source, IEqualityComparer<TSource> comparer = null)

HashSet<TSource>hashSet = new HashSet<TSource>(comparer);

foreach (TSource value in source)

if (hashSet.Add(value))

yield return value; // Deferred execution.

}

Union can be implemented by filtering the first source sequence with HashSet<T>.Add, then filter the second source sequence with HashSet<T>.Add:

public static IEnumerable<TSource> Union<TSource>(

this IEnumerable<TSource> first,

IEnumerable<TSource>second,

IEqualityComparer<TSource>comparer = null)

HashSet<TSource>hashSet = new HashSet<TSource>(comparer);

foreach (TSource firstValue in first)

if (hashSet.Add(firstValue))

yield return firstValue; // Deferred execution.

foreach (TSource secondValue in second)

if (hashSet.Add(secondValue))

yield return secondValue; // Deferred execution.

}

Except can be implemented with the same pattern of filtering with HashSet<T>.Add:

public static IEnumerable<TSource> Except<TSource>(

this IEnumerable<TSource> first,

IEnumerable<TSource>second,

IEqualityComparer<TSource>comparer = null)

HashSet<TSource>secondHashSet = new HashSet<TSource>(comparer).AddRange(second); // Eager evaluation.

foreach (TSource firstValue in first)

if (secondHashSet.Add(firstValue))

yield return firstValue; // Deferred execution.

}

When trying to pull the first value from the returned generator, values in the second sequence are eagerly evaluated and stored in a hash set, which is then used to filter the first sequence.

And Intersect can also be implemented with this pattern:

public static IEnumerable<TSource> IntersectWithAdd<TSource>(

this IEnumerable<TSource> first,

IEnumerable<TSource>second,

IEqualityComparer<TSource>comparer = null)

HashSet<TSource>secondHashSet = new HashSet<TSource>(comparer).AddRange(second); // Eager evaluation.

HashSet<TSource> firstHashSet = new HashSet<TSource>(comparer);

foreach (TSource firstValue in first)

if (secondHashSet.Add(firstValue))

firstHashSet.Add(firstValue);

else if (firstHashSet.Add(firstValue))

yield return firstValue; // Deferred execution.

}

To simplify above implementation, A Remove method can be defined for hash set:

public partial class HashSet<T>

public bool Remove(T value)

int hasCode = this.GetHashCode(value);

if (this.dictionary.ContainsKey(hasCode))

this.dictionary.Remove(hasCode);

return true;

return false;

}

Similar to Add, here if a value is found and removed, Remove outputs true; otherwise, Remove outputs false. So, Intersect can be implemented by filtering with Remove:

public static IEnumerable<TSource> Intersect<TSource>(

this IEnumerable<TSource> first,

IEnumerable<TSource>second,

IEqualityComparer<TSource>comparer = null)

HashSet<TSource>secondHashSet = new HashSet<TSource>(comparer).AddRange(second); // Eager evaluation.

foreach (TSource firstValue in first)

if (secondHashSet.Remove(firstValue))

yield return firstValue; // Deferred execution.

}

Convolution

Zip is easy to implement with a desugared foreach:

public static IEnumerable<TResult> Zip<TFirst, TSecond, TResult>(

this IEnumerable<TFirst> first,

IEnumerable<TSecond>second,

Func<TFirst, TSecond, TResult>resultSelector)

using (IEnumerator<TFirst> firstIterator = first.GetEnumerator())

using (IEnumerator<TSecond> secondIterator = second.GetEnumerator())

while (firstIterator.MoveNext() && secondIterator.MoveNext())

yield return resultSelector(firstIterator.Current, secondIterator.Current); // Deferred execution.

}

It stops yielding result when one of those 2 source sequences reaches the end.

Partitioning

Skip is easy to implement:

public static IEnumerable<TSource> Skip<TSource>(this IEnumerable<TSource> source, int count)

foreach (TSource value in source)

if (count > 0)

count--;

else

yield return value;

}

It can be optimized a little bit by desugaring the foreach loop, so that when a value should be skipped, only the source iterator’s MoveNext method is called.

public static IEnumerable<TSource> Skip<TSource>(this IEnumerable<TSource> source, int count)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (count > 0 && iterator.MoveNext())

count--; // Comparing foreach loop, iterator.Current is not called.

if (count <= 0)

while (iterator.MoveNext())

yield return iterator.Current; // Deferred execution.

}

In contrast, SkipWhile has to pull each value from source sequence to call predicate, so there is no need to desugar foreach. The following are the non-index overload and indexed overload:

public static IEnumerable<TSource> SkipWhile<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate)

bool skip = true;

foreach (TSource value in source)

if (skip && !predicate(value))

skip = false;

if (!skip)

yield return value; // Deferred execution.

public static IEnumerable<TSource> SkipWhile<TSource>(

this IEnumerable<TSource> source, Func<TSource, int, bool> predicate)

int index = -1;

bool skip = true;

foreach (TSource value in source)

index = checked(index + 1);

if (skip && !predicate(value, index))

skip = false;

if (!skip)

yield return value; // Deferred execution.

}

Take is also straightforward:

public static IEnumerable<TSource> Take<TSource>(this IEnumerable<TSource> source, int count)

if (count > 0)

foreach (TSource value in source)

yield return value; // Deferred execution.

if (--count == 0)

break;

}

And the following are TakeWhile’s non-indexed overload and indexed overload:

public static IEnumerable<TSource> TakeWhile<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate)

foreach (TSource value in source)

if (!predicate(value))

break;

yield return value; // Deferred execution.

public static IEnumerable<TSource> TakeWhile<TSource>(

this IEnumerable<TSource> source, Func<TSource, int, bool> predicate)

int index = -1;

foreach (TSource value in source)

index = checked(index + 1);

if (!predicate(value, index))

break;

yield return value; // Deferred execution.

}

Ordering

Reverse has been discussed:

public static IEnumerable<TSource> Reverse<TSource>(this IEnumerable<TSource> source)

TSource[] array = ToArray(source); // Eager evaluation.

for (int index = array.Length - 1; index >= 0; index--)

yield return array[index]; // Deferred execution.

}

The other ordering queries are implemented very differently because they involve the IOrderedEnumerable<T> interface. Again here are the signatures:

public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector, IComparer<TKey> comparer);

public static IOrderedEnumerable<TSource> OrderByDescending<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> OrderByDescending<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector, IComparer<TKey> comparer);

And once again the following is the definition of IOrderedEnumerable<T>:

namespace System.Linq

public interface IOrderedEnumerable<TElement> : IEnumerable<TElement>, IEnumerable

IOrderedEnumerable<TElement> CreateOrderedEnumerable<TKey>(

Func<TElement, TKey> keySelector, IComparer<TKey>comparer, bool descending);

}

Its implementation is a little complex. The following is a simplified version but equivalent to the built-in implementation:

internal class OrderedSequence<TSource, TKey> : IOrderedEnumerable<TSource>

private readonly IEnumerable<TSource> source;

private readonly IComparer<TKey> comparer;

private readonly bool descending;

private readonly Func<TSource, TKey> keySelector;

private readonly Func<TSource[], Func<int, int, int>> previousGetComparison;

internal OrderedSequence(

IEnumerable<TSource>source,

Func<TSource, TKey> keySelector,

IComparer<TKey>comparer,

bool descending = false,

// previousGetComparison is only specified in CreateOrderedEnumerable,

// and CreateOrderedEnumerable is only called by ThenBy/ThenByDescending.

// When OrderBy/OrderByDescending is called, previousGetComparison is not specified.

Func<TSource[], Func<int, int, int>> previousGetComparison = null)

this.source = source;

this.keySelector = keySelector;

this.comparer = comparer ?? Comparer<TKey>.Default;

this.descending = descending;

this.previousGetComparison = previousGetComparison;

public IEnumerator<TSource> GetEnumerator()

TSource[] values = this.source.ToArray(); // Eager evaluation.

int count = values.Length;

if (count <= 0)

yield break;

int[] indexMap = new int[count];

for (int index = 0; index < count; index++)

indexMap[index] = index;

// GetComparison is only called once for each generator instance.

Func<int, int, int> comparison = this.GetComparison(values);

Array.Sort(indexMap, (index1, index2) => // index1 < index2

// Format compareResult.

// When compareResult is 0 (equal), return index1 - index2,

// so that indexMap[index1] is before indexMap[index2],

// 2 equal values' original order is preserved.

int compareResult = comparison(index1, index2);

return compareResult == 0 ? index1 - index2 : compareResult;

}); // More eager evaluation.

for (int index = 0; index < count; index++)

yield return values[indexMap[index]];

IEnumerator IEnumerable.GetEnumerator() => this.GetEnumerator();

// Only called by ThenBy/ThenByDescending.

public IOrderedEnumerable<TSource> CreateOrderedEnumerable<TNextKey>

(Func<TSource, TNextKey> nextKeySelector, IComparer<TNextKey> nextComparer, bool nextDescending) =>

new OrderedSequence<TSource, TNextKey>(

this.source, nextKeySelector, nextComparer, nextDescending, this.GetComparison);

private TKey[] GetKeys(TSource[] values)

int count = values.Length;

TKey[] keys = new TKey[count];

for (int index = 0; index < count; index++)

keys[index] = this.keySelector(values[index]);

return keys;

private Func<int, int, int> GetComparison(TSource[] values)

// GetComparison is only called once for each generator instance,

// so GetKeys is only called once during the ordering query execution.

TKey[] keys = this.GetKeys(values);

if (this.previousGetComparison == null)

// In OrderBy/OrderByDescending.

return (index1, index2) =>

// OrderBy/OrderByDescending always need to compare keys of 2 values.

this.CompareKeys(keys, index1, index2);

// In ThenBy/ThenByDescending.

Func<int, int, int> previousComparison = this.previousGetComparison(values);

return (index1, index2) =>

// Only when previousCompareResult is 0 (equal),

// ThenBy/ThenByDescending needs to compare keys of 2 values.

int previousCompareResult = previousComparison(index1, index2);

return previousCompareResult == 0

? this.CompareKeys(keys, index1, index2)

: previousCompareResult;

};

private int CompareKeys(TKey[] keys, int index1, int index2)

// Format compareResult to always be 0, -1, or 1.

int compareResult = this.comparer.Compare(keys[index1], keys[index2]);

return compareResult == 0

? 0

: (this.descending ? (compareResult > 0 ? -1 : 1) : (compareResult > 0 ? 1 : -1));

}

Now the ordering queries can be implemented by constructing the above ordered sequence:

public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IComparer<TKey>comparer = null) =>

new OrderedSequence<TSource, TKey>(source, keySelector, comparer);

public static IOrderedEnumerable<TSource> OrderByDescending<TSource, TKey>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IComparer<TKey>comparer = null) =>

new OrderedSequence<TSource, TKey>(source, keySelector, comparer, descending: true);

public static IOrderedEnumerable<TSource> ThenBy<TSource, TKey>(

this IOrderedEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IComparer<TKey>comparer = null) =>

source.CreateOrderedEnumerable(keySelector, comparer, descending: false);

public static IOrderedEnumerable<TSource> ThenByDescending<TSource, TKey>(

this IOrderedEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

IComparer<TKey>comparer = null) =>

source.CreateOrderedEnumerable(keySelector, comparer, descending: true);

They implement deferred execution since constructing ordered sequence does not pull any value from the source. The OrderedSequence<T> type wraps the source data and iteration algorithm of ordering, including:

· the source sequence,

· the keySelector function,

· a bool value indicating the ordering should be descending or ascending

· a previousGetComparison function, which identifies whether current OrderedSequence<T> is created by OrderBy/OrderByDescending, or by ThenBy/ThenByDescending

o When OrderBy/OrderByDescending are called, they directly instantiate an OrderedSequence<T> with a null previousGetComparison function.

o When ThenBy/ThenByDescending are called, they call CreateOrderedEnumerable to instantiate OrderedSequence<T>, and pass its OrderedSequence<T>’s GetComparison method as the previousGetComparison function for the new OrderedSequence<T>.

OrderedSequence<T>’s GetEnumerator method uses yield statement to return an iterator (not generator this time). Eager evaluation is implemented, because it has to pull all values in the source sequence and sort them, in order to know which value is the first one to yield. For performance consideration, instead of sorting the values from source sequence, here the indexes of values are sorted. For example, in the values array, if indexes { 0, 1, 2 } become { 2, 0, 1 } after sorting, then the values are yielded in the order of { values[2], values[0], values[1] }.

When the eager evaluation starts, GetComparison is called. It evaluates all keys of the values, and returns a comparison function:

· If previousGetComparison function is null, it returns a comparison function to represent an OrderBy/OrderByDescending query, which just compares the keys.

· if previousGetComparison function is not null, it returns a comparison function to represent a ThenBy/ThenByDescending query, which first check the previous compare result, and only compare the keys when previous compare result is equal.

· In both cases, comparison function calls CompareKeys to compare 2 keys. CompareKeys calls IComparer<TKey>.Compare, and format the compare result to 0, -1, or 1 to represent less then, equal to, greater than. If the descending field is true, 1 and -1 are swapped.

Eventually, the returned comparison function is used during GetEnumerator’s eager evaluation, to sort the indexes of values. When comparing keys for index1 and index2, index1 is always less than index2. In another word, values[index1] is before values[index2] before the ordering query execution. If the result from comparison function is equal, index1 - index2 is used instead of 0. So that the relative positions of values at index1 and index2 are preserved, values[index1] is still before values[index2] after the ordering query execution.

Value queries

This category of queries iterate the source sequence. They apparently implement immediate execution.

Element

First can be implemented by pulling the source sequence once. The built-in implementation has some optimization. If the source already supports index, then source[0] can be pulled directly. The index support can be identified by testing if source also implements IList<T>:

namespace System.Collections.Generic

public interface IList<T> : ICollection<T>, IEnumerable<T>, IEnumerable

T this[int index] { get; set; }

int IndexOf(T item);

void Insert(int index, T item);

void RemoveAt(int index);

}

As fore mentioned, IList<T> is implemented by T[] array, List<T>, and Collection<T>, etc. So, the following is an optimized implementation of First:

public static TSource First<TSource>(this IEnumerable<TSource> source)

if (source is IList<TSource>list)

if (list.Count > 0)

return list[0];

else

foreach (TSource value in source)

return value;

throw new InvalidOperationException("Sequence contains no elements.");

}

The other overload with predicate is also easy to implement:

public static TSource First<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate)

foreach (TSource value in source)

if (predicate(value))

return value;

throw new InvalidOperationException("Sequence contains no matching element.");

}

The implementation of FirstOrDefault is very similar. When source is empty, just output the default value instead of throwing exception:

public static TSource FirstOrDefault<TSource>(this IEnumerable<TSource> source)

if (source is IList<TSource>list)

if (list.Count > 0)

return list[0];

else

foreach (TSource value in source)

return value;

return default;

public static TSource FirstOrDefault<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate)

foreach (TSource value in source)

if (predicate(value))

return value;

return default;

}

Last and LastOrDefault can be implemented in the similar pattern, with desugared foreach loop:

public static TSource Last<TSource>(this IEnumerable<TSource> source)

if (source is IList<TSource>list)

int count = list.Count;

if (count > 0)

return list[count - 1];

else

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (iterator.MoveNext())

TSource last;

do

last = iterator.Current;

while (iterator.MoveNext());

return last;

throw new InvalidOperationException("Sequence contains no elements.");

public static TSource Last<TSource>(this IEnumerable<TSource> source, Func<TSource, bool>predicate)

if (source is IList<TSource>list)

for (int index = list.Count - 1; index >= 0; index--)

TSource value = list[index];

if (predicate(value))

return value;

else

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (iterator.MoveNext())

TSource last = iterator.Current;

if (predicate(last))

while (iterator.MoveNext())

TSource value = iterator.Current;

if (predicate(value))

last = value;

return last;

throw new InvalidOperationException("Sequence contains no matching element.");

public static TSource LastOrDefault<TSource>(this IEnumerable<TSource> source)

if (source is IList<TSource>list)

int count = list.Count;

if (count > 0)

return list[count - 1];

else

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (iterator.MoveNext())

TSource last;

do

last = iterator.Current;

while (iterator.MoveNext());

return last;

return default;

public static TSource LastOrDefault<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate)

if (source is IList<TSource>list)

for (int index = list.Count - 1; index >= 0; index--)

TSource value = list[index];

if (predicate(value))

return value;

return default;

TSource last = default;

foreach (TSource value in source)

if (predicate(value))

last = value;

return last;

}

And ElementAt and ElementAtOrDefault too:

public static TSource ElementAt<TSource>(this IEnumerable<TSource> source, int index)

if (source is IList<TSource>list)

return list[index];

if (index < 0)

throw new ArgumentOutOfRangeException(nameof(index));

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (iterator.MoveNext())

if (index-- == 0)

return iterator.Current;

throw new ArgumentOutOfRangeException(nameof(index));

public static TSource ElementAtOrDefault<TSource>(this IEnumerable<TSource>source, int index)

if (index >= 0)

if (source is IList<TSource>list)

if (index < list.Count)

return list[index];

else

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (iterator.MoveNext())

if (index-- == 0)

return iterator.Current;

return default;

}

Single and SingleOrDefault are stricter:

public static TSource Single<TSource>(this IEnumerable<TSource> source)

if (source is IList<TSource>list)

switch (list.Count)

case 0:

throw new InvalidOperationException("Sequence contains no elements.");

case 1:

return list[0];

else

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (!iterator.MoveNext()) // source is empty.

throw new InvalidOperationException("Sequence contains no elements.");

TSource first = iterator.Current;

if (!iterator.MoveNext())

return first;

throw new InvalidOperationException("Sequence contains more than one element.");

public static TSource Single<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (iterator.MoveNext())

TSource value = iterator.Current;

if (predicate(value))

while (iterator.MoveNext())

if (predicate(iterator.Current))

throw new InvalidOperationException("Sequence contains more than one matching element.");

return value;

throw new InvalidOperationException("Sequence contains no matching element.");

public static TSource SingleOrDefault<TSource>(this IEnumerable<TSource>source)

if (source is IList<TSource>list)

switch (list.Count)

case 0:

return default;

case 1:

return list[0];

else

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (iterator.MoveNext())

TSource first = iterator.Current;

if (!iterator.MoveNext())

return first;

else

return default;

throw new InvalidOperationException("Sequence contains more than one element.");

public static TSource SingleOrDefault<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (iterator.MoveNext())

TSource value = iterator.Current;

if (predicate(value))

while (iterator.MoveNext())

if (predicate(iterator.Current))

throw new InvalidOperationException("Sequence contains more than one matching element.");

return value;

return default;

}

Aggregation

Aggregate pulls all values from source and accumulate them:

public static TResult Aggregate<TSource, TAccumulate, TResult>(

this IEnumerable<TSource> source,

TAccumulate seed,

Func<TAccumulate, TSource, TAccumulate>func,

Func<TAccumulate, TResult> resultSelector)

TAccumulate accumulate = seed;

foreach (TSource value in source)

accumulate = func(accumulate, value);

return resultSelector(accumulate);

public static TAccumulate Aggregate<TSource, TAccumulate>(

this IEnumerable<TSource> source, TAccumulate seed, Func<TAccumulate, TSource, TAccumulate> func)

TAccumulate accumulate = seed;

foreach (TSource value in source)

accumulate = func(accumulate, value);

return accumulate;

public static TSource Aggregate<TSource>(

this IEnumerable<TSource> source, Func<TSource, TSource, TSource> func)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (!iterator.MoveNext())

throw new InvalidOperationException("Sequence contains no elements.");

TSource accumulate = iterator.Current;

while (iterator.MoveNext())

accumulate = func(accumulate, iterator.Current);

return accumulate;

}

Count can be implemented by iterating the source sequence. And if the source sequence is a collection, then it has a Count property:

public static int Count<TSource>(this IEnumerable<TSource> source)

switch (source)

case ICollection<TSource> genericCollection:

return genericCollection.Count;

case ICollection collection:

return collection.Count;

default:

int count = 0;

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (iterator.MoveNext())

count = checked(count + 1); // Comparing foreach loop, iterator.Current is never called.

return count;

}

And the overload with predicate can be implemented by filtering with the predicate function:

public static int Count<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate)

int count = 0;

foreach (TSource value in source)

if (predicate(value))

count = checked(count + 1);

return count;

}

LongCount cannot use collections’ Count property because it has int output. It simply counts the values:

public static long LongCount<TSource>(this IEnumerable<TSource> source)

long count = 0L;

using (IEnumerator<TSource> iterator = source.GetEnumerator())

while (iterator.MoveNext())

count = checked(count + 1L); // Comparing foreach loop, iterator.Current is never called.

return count;

public static long LongCount<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate)

long count = 0L;

foreach (TSource value in source)

if (predicate(value))

count = checked(count + 1L);

return count;

}

Regarding the naming of LongCount .NET Framework Design Guidelines’ General Naming Conventions says:

DO use a generic CLR type name, rather than a language-specific name.

It would be more consistent if LongCount was named as Int64Count, just like Convert.ToInt64, etc.

Min has 22 overloads; the following is the overload for decimal:

public static decimal Min(this IEnumerable<decimal> source)

decimal min;

using (IEnumerator<decimal> iterator = source.GetEnumerator())

if (!iterator.MoveNext())

throw new InvalidOperationException("Sequence contains no elements.");

min = iterator.Current;

while (iterator.MoveNext())

decimal value = iterator.Current;

if (value < min)

min = value;

return min;

}

And the decimal overload with selector can be implemented with Select:

public static decimal Min<TSource>(

this IEnumerable<TSource> source, Func<TSource, decimal>selector) => source.Select(selector).Min();

Max also has 22 overloads. The overload for decimal without and with selector can be implemented with the same pattern:

public static decimal Max(this IEnumerable<decimal> source)

decimal max;

using (IEnumerator<decimal> iterator = source.GetEnumerator())

if (!iterator.MoveNext())

throw new InvalidOperationException("Sequence contains no elements.");

max = iterator.Current;

while (iterator.MoveNext())

decimal value = iterator.Current;

if (value > max)

max = value;

return max;

public static decimal Max<TSource>(

this IEnumerable<TSource> source, Func<TSource, decimal>selector) => source.Select(selector).Max();

Sum/Average has 20 overloads each. Also take the decimal overloads as example:

public static decimal Sum(this IEnumerable<decimal> source)

decimal sum = 0;

foreach (decimal value in source)

sum += value;

return sum;

public static decimal Sum<TSource>(this IEnumerable<TSource> source, Func<TSource, decimal> selector) =>

source.Select(selector).Sum();

public static decimal Average<TSource>(this IEnumerable<TSource>source, Func<TSource, decimal> selector)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

if (!iterator.MoveNext())

throw new InvalidOperationException("Sequence contains no elements.");

decimal sum = selector(iterator.Current);

long count = 1L;

while (iterator.MoveNext())

sum += selector(iterator.Current);

count++;

return sum / count;

}

Quantifier

All, Any, and Contains has a bool output. They can be implemented in a similar foreach-if pattern:

public static bool All<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate)

foreach (TSource value in source)

if (!predicate(value))

return false;

return true;

public static bool Any<TSource>(this IEnumerable<TSource>source, Func<TSource, bool> predicate)

foreach (TSource value in source)

if (predicate(value))

return true;

return false;

public static bool Any<TSource>(this IEnumerable<TSource>source)

using (IEnumerator<TSource> iterator = source.GetEnumerator())

return iterator.MoveNext(); // Not needed to call iterator.Current.

public static bool Contains<TSource>(

this IEnumerable<TSource> source,

TSource value,

IEqualityComparer<TSource>comparer = null)

if (comparer == null &&source is ICollection<TSource> collection)

return collection.Contains(value);

comparer = comparer ?? EqualityComparer<TSource>.Default;

foreach (TSource sourceValue in source)

if (comparer.Equals(sourceValue, value))

return true;

return false;

}

Contains can be optimized a little bit because collection already has a Contains method.

Equality

The implementation of SequenceEqual is a little similar to Zip, where 2 source sequences are iterated at the same time. They are equal only when their counts are equal, and their values at each index are equal:

public static bool SequenceEqual<TSource>(

this IEnumerable<TSource> first,

IEnumerable<TSource>second,

IEqualityComparer<TSource>comparer = null)

comparer = comparer ?? EqualityComparer<TSource>.Default;

if (first is ICollection<TSource> firstCollection && second is ICollection<TSource>secondCollection

&& firstCollection.Count != secondCollection.Count)

return false;

using (IEnumerator<TSource> firstIterator = first.GetEnumerator())

using (IEnumerator<TSource> secondIterator = second.GetEnumerator())

while (firstIterator.MoveNext())

if (!secondIterator.MoveNext() || !comparer.Equals(firstIterator.Current, secondIterator.Current))

return false;

return !secondIterator.MoveNext();

}

Summary

This chapter discusses iterator pattern, and its implementation with the yield syntactic sugar. In LINQ to Objects, collection queries and value queries implements immediate execution, and sequence queries implements deferred execution following the iterator pattern, which is either lazy evaluation, or eager evaluation. Based on these concepts, this chapter demonstrates the implementation of LINQ to Objects queries in an equivalent and readable way. With the understanding of queries’ internal implementation, you should have a good understanding of LINQ to Objects standard queries, and be able to use them effectively.

LINQ to Objects in Depth (4) Deferred Execution, Lazy Evaluation and Eager Evaluation

Thu, 04 Jul 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to Objects in Depth series]

As fore mentioned, when LINQ to Objects’ collection queries and value queries are called, they start to evaluate query result. When sequence queries are called, they do not evaluate any query result, and can be viewed as defining a query.

Immediate execution vs. deferred execution

Apparently, collection queries must pull all values from source sequence, store them to a collection in the specified way, and output the collection to caller. The value queries also must pull or detect values in the source sequence to have the query result available. This is called immediate execution of LINQ queries. These queries mutate their source sequence’s iterator state, and may cause side effect, so they are impure functions. The sequence queries are pure functions, implemented by following the above iteration pattern. So, when they are called, they only construct a generator. This is called deferred execution.

Generally, for LINQ query or any function with a sequence output, the yield syntactic sugar is a very handy tool to implement deferred execution. Tis can be intuitively demonstrated by the following 2 functions:

internal static IEnumerable<double> AbsAndSqrtArray(double @double) =>

new double[]

Math.Abs(@double),

Math.Sqrt(@double)

};

internal static IEnumerable<double> AbsAndSqrtGenerator(double @double)

yield return Math.Abs(@double);

yield return Math.Sqrt(@double);

}

When the first function AbsAndSqrtArray is called, Math.Abs and Math.Sqrt are called immediately to evaluate 2 values, and these 2 values are stored in an array for output. To defer the execution of the Math.Abs and Math.Sqrt, the second function AbsAndSqrtGenerator uses the yield syntactic sugar. When it is called, it constructs a generator for output. Only when pulling the 2 values from the output generator, Math.Abs and Math.Sqrt are called.

The first function’s output sequence is a collection with actual values evaluated and stored, sometimes it is called a hot IEnumerable<T>, and the second function’s output sequence is called a cold IEnumerable<T>. Apparently, in LINQ to Objects, all sequence queries’ output are cold IEnumerable<T>.

Lazy evaluation vs. eager evaluation

The Select and Where queries’ execution is deferred until values are pulled from the result sequence. When pulling the first result, the query executes, the selector function and predicate function are called until the first result is evaluated. At this moment the rest results are not evaluated. When pulling the second result, the query executes until the second result is evaluated, and so on. If pulling stops in the middle, the rest results are not evaluated. This behaviour is called lazy evaluation. To demonstrate the evaluation, add tracing to Select query’s iteration control flow:

internal static IEnumerable<TResult> SelectGenerator<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector)

"Select query starts.".WriteLine();

foreach (TSource value in source)

$"Select query calls selector with {value}.".WriteLine();

TResult result = selector(value);

$"Select query yields {result}.".WriteLine();

yield return result;

"Select query ends.".WriteLine();

}

The above foreach statement can be desugared as the following actual control flow:

internal static IEnumerable<TResult> DesugaredSelectGenerator<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector)

"Select query starts.".WriteLine();

IEnumerator<TSource> sourceIterator = null; // start.

try

sourceIterator = source.GetEnumerator(); // start.

while (sourceIterator.MoveNext()) // moveNext.

$"Select query calls selector with {sourceIterator.Current}.".WriteLine(); // getCurrent.

TResult result = selector(sourceIterator.Current); // getCurrent.

$"Select query yields {result}.".WriteLine(); // getCurrent.

yield return result; // getCurrent.

finally

sourceIterator?.Dispose(); // dispose.

"Select query ends.".WriteLine(); // end.

}

Its compilation is equivalent to the following generator construction:

internal static IEnumerable<TResult> CompiledSelectGenerator<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector)

IEnumerator<TSource> sourceIterator = null;

return new Generator<TResult>(

start: () =>

"Select query starts.".WriteLine();

sourceIterator = source.GetEnumerator();

},

moveNext: () => sourceIterator.MoveNext(),

getCurrent: () =>

$"Select query calls selector with {sourceIterator.Current}.".WriteLine();

TResult result = selector(sourceIterator.Current);

$"Select query yields {result}.".WriteLine();

return result;

},

dispose: () => sourceIterator?.Dispose(),

end: () => "Select query ends.".WriteLine());

}

Similarly, add tracing to Where query as well:

internal static IEnumerable<TSource> WhereGenerator<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> predicate)

"Where query starts.".WriteLine();

foreach (TSource value in source)

$"Where query calls predicate with {value}.".WriteLine();

if (predicate(value))

$"Where query yields {value}.".WriteLine();

yield return value;

"Where query ends.".WriteLine();

}

Its compilation is also a generator construction, equivalent to the following:

internal static IEnumerable<TSource> CompiledWhereGenerator<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> predicate)

IEnumerator<TSource> sourceIterator = null;

return new Generator<TSource>(

start: () =>

"Where query starts.".WriteLine();

sourceIterator = source.GetEnumerator();

},

moveNext: () =>

while (sourceIterator.MoveNext())

$"Where query calls predicate with {sourceIterator.Current}."

.WriteLine();

if (predicate(sourceIterator.Current))

return true;

return false;

},

getCurrent: () =>

$"Where query yields {sourceIterator.Current}.".WriteLine();

return sourceIterator.Current;

},

dispose: () => sourceIterator?.Dispose(),

end: () => "Where query ends.".WriteLine());

}

The following LINQ query is a composition of Where and Select:

internal static void CallWhereAndSelect()

IEnumerable<int> source = Enumerable.Range(1, 5);

IEnumerable<string> deferredQuery = source)

.WhereGenerator(int32 => int32 > 2) // Deferred execution.

.SelectGenerator(int32 => new string('*', int32)); // Deferred execution.

foreach (string result in deferredQuery)

// Select query starts.

// Where query starts.

// Where query calls predicate with 1.

// Where query calls predicate with 2.

// Where query calls predicate with 3.

// Where query yields 3.

// Select query calls selector with 3.

// Select query yields ***.

// Where query calls predicate with 4.

// Where query yields 4.

// Select query calls selector with 4.

// Select query yields ****.

// Where query calls predicate with 5.

// Where query yields 5.

// Select query calls selector with 5.

// Select query yields *****.

// Where query ends.

// Select query ends.

}

The tracing messages shows how exactly the lazy evaluation works in deferred execution. When WhereGenerator and SelectGenerator are called, the mapping query and filtering query are not executed. The composited query is the output sequence of SelectGenerator. When pulling the first result, Select starts execution, Select pulls the first results from its source sequence, which is the output sequence of Where, so that Where query starts execution, and pulls its source sequence. Where first retrieves 1, and calls its predicate function with 1. In this case, the output of predicate is false, so Where does not yield 1, but pulls the next value 2 from its source sequence, and calls predicate with 2. Again, the output of predicate is false, so Where does not yield 2, but pulls the next value 3 from its source sequence. This time, the output of predicate function is true, so Where query yields 3 to Select query. Select retrieves 3 and calls its selector function with 3. Eventually Select yields selector output *** as the first result of composited query. Then Select query pulls the next value from Where query, Where query pulls the next value from source. Where query retrieves 4 and calls predicate with 4. The output of predicate is true, so Where query yields 4 to Select query. Select query retrieves 4 and calls selector with 4. Eventually Select yields selector output **** as the second result of the composited query, and so on.

The opposition of lazy evaluation is eager evaluation, where pulling the first result causes all results being evaluated. For example, Reverse query is a sequence query, and it implements deferred execution. When its result sequence is pulled for the first time, it starts execution. It has to evaluate and store all the values of source sequence, in order to yields the last source value as the first query result. The following is the equivalent implementation of Reverse. Again, to demonstrate the evaluation, tracing is added as well:

internal static IEnumerable<TSource> ReverseGenerator<TSource>(

this IEnumerable<TSource> source)

"Reverse query starts.".WriteLine();

TSource[] results = source.ToArray();

$"Reverse query has all {results.Length} value(s) of source sequence.".WriteLine();

for (int index = results.Length - 1; index >= 0; index--)

$"Reverse query yields index {index} of source sequence.".WriteLine();

yield return results[index];

"Reverse query ends.".WriteLine();

}

Its compilation is equivalent to the following generator construction:

internal static IEnumerable<TSource> CompiledReverseGenerator<TSource>(this IEnumerable<TSource> source)

TSource[] results = null;

int index = 0;

return new Generator<TSource>(

start: () =>

"Reverse query starts.".WriteLine();

results = source.ToArray();

$"Reverse query evaluated all {results.Length} value(s) of source sequence.".WriteLine();

index = results.Length - 1;

},

moveNext: () => index >= 0,

getCurrent: () =>

$"Reverse query yields index {index} of source source.".WriteLine();

return results[index--];

},

end: () => "Reverse query ends.".WriteLine());

}

The following example is the composition of Select query and Reverse query:

internal static void CallSelectAndReverse()

IEnumerable<string> deferredQuery = Enumerable.Range(1, 5)

.SelectGenerator(int32 => new string('*', int32)) // Deferred execution.

.ReverseGenerator(); // Deferred execution.

foreach (string result in deferredQuery)

// Reverse query starts.

// Select query starts.

// Select query calls selector with 1.

// Select query yields *.

// Select query calls selector with 2.

// Select query yields **.

// Select query calls selector with 3.

// Select query yields ***.

// Select query calls selector with 4.

// Select query yields ****.

// Select query calls selector with 5.

// Select query yields *****.

// Select query ends.

// Reverse query has all 5 value(s) of source sequence.

// Reverse query yields index 4 of source sequence.

// Reverse query yields index 3 of source sequence.

// Reverse query yields index 2 of source sequence.

// Reverse query yields index 1 of source sequence.

// Reverse query yields index 0 of source sequence.

// Reverse query ends.

}

The tracing messages shows how exactly the eager evaluation works in deferred execution. When pulling the first result, Reverse starts execution. Reverse pulls all values from its source sequence, which is the output generator of Select, so that Select query starts execution. Select pulls all values from its source sequence store them, calls selector function, and yields all results to Reverse query. Reverse retrieves and stores all values, and yields the last value as the first result. Then pulling the next query result causes Reverse direct yield the stored second last value, and so on.

LINQ to Objects in Depth (3) Generator

Wed, 03 Jul 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to Objects in Depth series]

After understanding how to use LINQ to Objects queries, this chapter starts to discuss how these queries work internally, including how they execute and how they are implemented. These insights help developer mastering LINQ to Objects.

Generator

Most LINQ to Objects queries has an IEnumerable<T> output. They can be easily implemented with “brute force”. Take the simple query Repeat as example:

internal static IEnumerable<TSource> RepeatArray<TSource>(TSource value, int count)

TSource[] results = new TSource[count];

for (int index = 0; index < count; index++)

results[index] = value;

return results;

}

The problem is, when the above Repeat query is called, it immediately generates all query results, stores them in an array, and outputs to the caller. When the count argument is large, this implementation consumes large memory. This can be resolved by properly following the iterator pattern.

Implement iterator pattern

The iterator pattern is imperative and object-oriented. It has a sequence represented by IEnumerable<T>, and an iterator represented by IEnumerator<T>. According to the definition of IEnumerator<T>, apparently the iterator is stateful. The following is a general-purpose iterator implemented as a finite-state machine:

public enum IteratorState

Create = -2,

Start = 0,

MoveNext = 1,

End = -1,

Error = -3

public class Iterator<T> : IEnumerator<T>

protected readonly Action start;

protected readonly Func<bool> moveNext;

protected readonly Func<T> getCurrent;

protected readonly Action dispose;

protected readonly Action end;

public Iterator(

Action start = null,

Func<bool> moveNext = null,

Func<T> getCurrent = null,

Action dispose = null,

Action end = null)

this.start = start;

this.moveNext = moveNext;

this.getCurrent = getCurrent;

this.dispose = dispose;

this.end = end;

public T Current { get; private set; }

object IEnumerator.Current => this.Current;

internal IteratorState State { get; private set; } = IteratorState.Create; // IteratorState: Create.

internal Iterator<T> Start()

this.State = IteratorState.Start; // IteratorState: Create => Start.

return this;

public bool MoveNext()

try

switch (this.State)

case IteratorState.Start:

this.start?.Invoke();

this.State = IteratorState.MoveNext; // IteratorState: Start => MoveNext.

goto case IteratorState.MoveNext;

case IteratorState.MoveNext:

if (this.moveNext?.Invoke() ?? false)

this.Current = this.getCurrent != null ? this.getCurrent() : default;

return true; // IteratorState: MoveNext => MoveNext.

this.State = IteratorState.End; // IteratorState: MoveNext => End.

this.dispose?.Invoke();

this.end?.Invoke();

break;

return false;

catch

this.State = IteratorState.Error; // IteratorState: Start, MoveNext, End => Error.

this.Dispose();

throw;

public void Dispose()

if (this.State == IteratorState.Error || this.State == IteratorState.MoveNext)

try { }

finally

// Unexecuted finally blocks are executed before the thread is aborted.

this.State = IteratorState.End; // IteratorState: Error => End.

this.dispose?.Invoke();

public void Reset() => throw new NotSupportedException();

}

The sequence can be simply viewed as an iterator factory:

public class Sequence<T> : IEnumerable<T>

private readonly Func<Iterator<T>> iteratorFactory;

public Sequence(Func<Iterator<T>> iteratorFactory) =>

this.iteratorFactory = iteratorFactory;

public IEnumerator<T> GetEnumerator() =>

this.iteratorFactory().Start(); // IteratorState: Create => Start.

IEnumerator IEnumerable.GetEnumerator() => this.GetEnumerator();

}

The above iterator encapsulates 5 functions (start, moveNext, getCurrent, end, dispose) in the iteration control flow, and manages 5 states:

· Create: if an iterator is constructed on the fly, its initial state is Create.

· Start: if an iterator is created by sequence’s factory method, its state is Start. Later, if its MoveNext is called for the first time, the start function is called to do the initialization work. Then, the state changes to MoveNext

· MoveNext: After its MoveNext method being called for the first time, its state is MoveNext. Each time its MoveNext method is called, the moveNext function is called to output a bool value

o If the output is true, the iterator has a value available for the caller, and getCurrent function can be called through its Current property to pull that value; The state remains MoveNext.

o If false, there is no value available. The state changes to End, and the dispose function is called to release resources, then end functions is called to do the cleanup work;

· End: if its MoveNext method is called and the state is End, false is directly returned to indicate caller that the sequential traversal ended, there is no value available to pull.

· Error: if its MoveNext method throws an exception, the state changes to Error. Then its Dispose method is called to do the cleanup work, and eventually its state is changed to End.

Now Sequence<T> and Iterator<T> can be used to implement Repeat with improved performance:

internal static IEnumerable<TSource> RepeatSequence<TSource>(

TSource value, int count) =>

new Sequence<TSource>(() =>

int index = 0;

return new Iterator<TSource>(

moveNext: () => index++ < count,

getCurrent: () => value);

});

When the above RepeatSequence is called, it does not generate and store all repeated values. It only constructs a sequence instance with an iterator factory function, which then can be consumed by caller with a foreach statement:

internal static void CallRepeatSequence<TSource>(TSource value, int count)

foreach (TSource result in RepeatSequence(value, count)) { }

// Compiled to:

using (IEnumerator<TSource> iterator = RepeatSequence(value, count).GetEnumerator())

while (iterator.MoveNext())

TSource result = iterator.Current;

// Virtual control flow inside iterator:

int index = 0;

try

while (index++ < count)

TSource result = value;

finally { }

}

The foreach statement is compiled to a call to the sequence’s GetEnumerator method to get an iterator, and a while loop to poll iterator’s MoveNext method and Current property getter. The iterator’s state is an int value to indicate current result’ index in all results. When MoveNext is called, it checks and mutates the state. If the state is valid, a result is available be pulled from Current. With iterator pattern, RepeatSequence is improved from RepeatArray. The results now are on-demand, they are not generated and stored in advance.

Another example is Select query. It accepts an input source sequence and constructs a new result sequence, by mapping each source value to another result with a selector function. It can be implemented following iterator pattern:

internal static IEnumerable<TResult> SelectSequence<TSource, TResult>(

IEnumerable<TSource> source, Func<TSource, TResult> selector) =>

new Sequence<TResult>(() =>

IEnumerator<TSource> sourceIterator = null;

return new Iterator<TResult>(

start: () => sourceIterator = source.GetEnumerator(),

moveNext: () => sourceIterator.MoveNext(),

getCurrent: () => selector(sourceIterator.Current),

dispose: () => sourceIterator?.Dispose());

});

So that, Select dos not map or store any query result. It only constructs a sequence and never calls the selector function. When caller pulls each result from the output sequence, the selector function is called and result is evaluated:

internal static void CallSelectSequence<TSource, TResult>(

IEnumerable<TSource> source, Func<TSource, TResult> selector)

foreach (TResult result in SelectSequence(source, selector)) { }

// Compiled to:

using (IEnumerator<TResult> iterator = SelectSequence(source, selector).GetEnumerator())

while (iterator.MoveNext())

TResult result = iterator.Current;

// Virtual control flow inside iterator:

IEnumerator<TSource> sourceIterator = null;

try

sourceIterator = source.GetEnumerator();

while (sourceIterator.MoveNext())

TResult result = selector(sourceIterator.Current);

finally

sourceIterator?.Dispose();

}

Similarly, Where can be implemented with this pattern to filter the source sequence with a predicate function:

internal static IEnumerable<TSource> WhereSequence<TSource>(

IEnumerable<TSource> source, Func<TSource, bool> predicate) =>

new Sequence<TSource>(() =>

IEnumerator<TSource> sourceIterator = null;

return new Iterator<TSource>(

start: () => sourceIterator = source.GetEnumerator(),

moveNext: () =>

while (sourceIterator.MoveNext())

if (predicate(sourceIterator.Current))

return true;

return false;

},

getCurrent: () => sourceIterator.Current,

dispose: () => sourceIterator?.Dispose());

});

Again, Where does not filter the source or store the filtering results. It only constructs a sequence and never calls predicate function. When caller starts to pull a result from the output sequence, the predicate function is called:

internal static void CallWhereSequence<TSource>(

IEnumerable<TSource> source, Func<TSource, bool> predicate)

foreach (TSource result in WhereSequence(source, predicate)) { }

// Compiled to:

using (IEnumerator<TSource> iterator = WhereSequence(source, predicate).GetEnumerator())

while (iterator.MoveNext())

TSource result = iterator.Current;

// Virtual control flow inside iterator:

IEnumerator<TSource> sourceIterator = null;

try

sourceIterator = source.GetEnumerator();

while (sourceIterator.MoveNext())

if (predicate(sourceIterator.Current))

TSource result = sourceIterator.Current;

finally

sourceIterator?.Dispose();

}

With sequence and iterator, the LINQ sequential queries can be correctly implemented with expected performance. However, since the iterator pattern is imperative and stateful, it is not straightforward to specify how to construct sequence and how to construct iterator, and the actual iteration workflow is not intuitive as well. Another example is to define a custom FromValue query to generate a sequence from a single value, following the iterator pattern:

internal static IEnumerable<TSource> FromValueSequence<TSource>(TSource value) =>

new Sequence<TSource>(() =>

bool isValueIterated = false;

return new Iterator<TSource>(

moveNext: () =>

if (!isValueIterated)

isValueIterated = true;

return true;

return false;

},

getCurrent: () => value);

});

It uses a bool value as iterator state, so that the value cam be pulled only once:

internal static void CallFromValueSequence<TSource>(TSource value)

foreach (TSource result in FromValueSequence(value)) { }

// Compiled to:

using (IEnumerator<TSource> iterator = FromValueSequence(value).GetEnumerator())

while (iterator.MoveNext())

TSource result = iterator.Current;

// Virtual control flow inside iterator:

bool isValueIterated = false;

try

while (!isValueIterated)

isValueIterated = true;

TSource result = value;

finally { }

}

To simplify the coding, C# provides yield statement.

yield statement

C# 2.0 introduces the yield statement for named functions to implement iterator pattern, without defining and constructing sequence and iterator. The following example is equivalent to above RepeatSequence:

internal static IEnumerable<TSource> RepeatYield<TSource>(TSource value, int count)

// Virtual control flow when iterating the results:

// int index = 0;

// try

// {

// while (index++ < count)

// {

// TSource result = value;

// }

// }

// finally { }

int index = 0;

try

while (index++ < count)

yield return value;

finally { }

}

With the yield statement, the start, moveNext, getCurrent, end, dispose functions for iteration are merged into a fluent and intuitive control flow. The yield statement is a syntactic sugar. The compilation of RepeatYield is just slightly different from the previous RepeatSequence implementation with the standard iteration pattern. C# compiler generates a generator type for each named function with yield statement and IEnumerable/IEnumerable<T> output. A generator is both a sequence and an iterator, which can be virtually viewed as:

public interface IGenerator<out T> : IEnumerable<T>, IEnumerator<T> { }

In another word, a generator is an iterator with an additional factory method GetEnumerator, which does a little more work:

public class Generator<T> : Iterator<T>, IGenerator<T>

private readonly int initialThreadId = Environment.CurrentManagedThreadId;

public Generator(

Action start = null,

Func<bool> moveNext = null,

Func<T> getCurrent = null,

Action dispose = null,

Action end = null) : base(start, moveNext, getCurrent, dispose, end)

{ }

public IEnumerator<T> GetEnumerator() =>

this.initialThreadId == Environment.CurrentManagedThreadId

&& this.State == IteratorState.Create

// Called by the same initial thread and iteration is not started.

? this.Start()

// If the iteration is already started, or the iteration is requested from a different thread, create new generator with new iterator.

: new Generator<T>(this.start, this.moveNext, this.getCurrent, this.dispose, this.end).Start();

IEnumerator IEnumerable.GetEnumerator() => this.GetEnumerator();

}

When generator is constructed, it remembers the thread id. Later, if its factory method is called by the same thread and the generator’s state has not been mutated, the output is itself; if the its factory is called again by the same thread after its state is mutated, or called by another thread, the output is a new generator. So, the compilation of RepeatYield is equivalent to:

internal static IEnumerable<TSource> RepeatGenerator<TSource>(TSource value, int count)

int index = 0;

return new Generator<TSource>(

moveNext: () => index++ < count,

getCurrent: () => value);

}

The control flow with yield statement can be viewed as normal control flow, so the above control flows can be further simplified, with the same way of simplifying normal C# code. Here the try-finally statement is not needed, so can be removed. And the while loop is equivalent to the following for loop:

internal static IEnumerable<TSource> Repeat<TSource>(TSource value, int count)

for (int index = 0; index < count; index++)

yield return value;

}

So, yield statement greatly simplifies the implementation of iterator pattern. Again, the above for loop is a virtual control flow. At compile time, a generator type is generated to do the real work, and Repeat is compiled to just construct and output a generator. At runtime, when Repeat is called, the for loop is not executed and no query result is evaluated or stored; only when the caller pulls result, the generator yields repeated query results with the specified count.

Similarly, Select and Where standard queries can also be implemented with yield statement following their iteration control flows:

internal static IEnumerable<TResult> SelectYield<TSource, TResult>(

IEnumerable<TSource> source, Func<TSource, TResult> selector)

IEnumerator<TSource> sourceIterator = null;

try

sourceIterator = source.GetEnumerator(); // start.

while (sourceIterator.MoveNext()) // moveNext.

yield return selector(sourceIterator.Current); // getCurrent.

finally

sourceIterator?.Dispose(); // dispose.

// Compiled to:

// IEnumerator<TSource> sourceIterator = null;

// return new Generator<TResult>(

// start: () => sourceIterator = source.GetEnumerator(),

// moveNext: () => sourceIterator.MoveNext(),

// getCurrent: () => selector(sourceIterator.Current),

// dispose: () => sourceIterator?.Dispose());

internal static IEnumerable<TSource> WhereYield<TSource>(

IEnumerable<TSource> source, Func<TSource, bool> predicate)

IEnumerator<TSource> sourceIterator = null;

try

sourceIterator = source.GetEnumerator(); // start.

while (sourceIterator.MoveNext()) // moveNext.

if (predicate(sourceIterator.Current)) // moveNext.

yield return sourceIterator.Current; // getCurrent.

finally

sourceIterator?.Dispose(); // dispose.

// Compiled to:

// IEnumerator<TSource> sourceIterator = null;

// return new Generator<TSource>(

// start: () => sourceIterator = source.GetEnumerator(),

// moveNext: () =>

// {

// while (sourceIterator.MoveNext())

// {

// if (predicate(sourceIterator.Current))

// {

// return true;

// }

// }

//

// return false;

// },

// getCurrent: () => sourceIterator.Current,

// dispose: () => sourceIterator?.Dispose());

}

These control flows can be simplified with a foreach statement:

internal static IEnumerable<TResult> Select<TSource, TResult>(

IEnumerable<TSource> source, Func<TSource, TResult> selector)

foreach (TSource value in source)

yield return selector(value);

internal static IEnumerable<TSource> Where<TSource>(

IEnumerable<TSource> source, Func<TSource, bool> predicate)

foreach (TSource value in source)

if (predicate(value))

yield return value;

}

The FromValue custom query can be implemented with yield statement too:

internal static IEnumerable<TSource> FromValueYield<TSource>(TSource value)

bool isValueIterated = false;

try

while (!isValueIterated) // moveNext.

isValueIterated = true; // moveNext.

yield return value; // getCurrent.

finally { }

// Compiled to:

// bool isValueIterated = false;

// return new Generator<TSource>(

// moveNext: () =>

// {

// while (!isValueIterated)

// {

// isValueIterated = true;

// return true;

// }

//

// return false;

// },

// getCurrent: () => value);

}

The above try-finally statement, state checking and mutation are not needed. It is the same control flow as the following:

internal static IEnumerable<TSource> FromValue<TSource>(TSource value)

yield return value;

}

A named function with yield statement must have an IEnumerable/IEnumerable<T> output, or an IEnumerator/IEnumerator<T> output. As demonstrated above, when it outputs a sequence, it is compiled to generator construction. When it outputs an iterator, it is compiled to iterator construction. Take Repeat as example, its output can also be IEnumerator<T>:

internal static IEnumerator<TSource> RepeatIterator<TSource>(TSource value, int count)

for (int index = 0; index < count; index++)

yield return value;

// Compiled to:

// int index = 0;

// return new Iterator<TSource>(

// moveNext: () => index++ < count,

// getCurrent: () => value).Start();

internal static void CallRepeatIterator<TSource>(TSource value, int count)

using (IEnumerator<TSource> iterator = RepeatIterator(value, count))

while (iterator.MoveNext())

TSource result = iterator.Current;

}

The yield statement has another form as yield break, which means to end the iteration. For example, the following function outputs a sequence of 0, 1, 2:

internal static IEnumerable<int> YieldBreak()

yield return 1;

yield return 2;

yield return 3;

yield break;

yield return 4;

}

The Repeat query can be implemented with the following equivalent control flow and yield break:

internal static IEnumerable<TSource> RepeatYieldBreak<TSource>(TSource value, int count)

int index = 0;

while (true)

if (index++ < count)

yield return value;

else

yield break;

}

LINQ to Objects in Depth (2) Query Methods (Operators) and Query Expressions

Tue, 02 Jul 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to Objects in Depth series]

As fore mentioned, LINQ to Objects standard query methods (also called standard query operators) are provided as static methods of System.Linq.Enumerable type, most of which are IEnumerable<T> extension methods. They can be categorized by output type:

Sequence queries: output a new IEnumerable<T> sequence:

o Generation: Empty , Range, Repeat, DefaultIfEmpty

o Filtering (restriction): Where*, OfType

o Mapping (projection): Select*, SelectMany*

o Grouping: GroupBy*

o Join: SelectMany, Join*, GroupJoin*

o Concatenation: Concat, Append, Prepend

o Set: Distinct, Union, Intersect, Except

o Convolution: Zip

o Partitioning: Take, Skip, TakeWhile, SkipWhile

o Ordering: OrderBy*, ThenBy*, OrderByDescending*, ThenByDescending*, Reverse*

o Conversion: Cast*, AsEnumerable

Collection queries: output a new collection:

o Conversion: ToArray, ToList, ToDictionary, ToLookup

Value queries: output a single value:

o Element: First, FirstOrDefault, Last, LastOrDefault, ElementAt, ElementAtOrDefault, Single, SingleOrDefault

o Aggregation: Aggregate, Count, LongCount, Min, Max, Sum, Average

o Quantifier: All, Any, Contains

o Equality: SequenceEqual

These LINQ query methods are very functional. They are functions that can be composed by fluent chaining. Many of them are higher-order functions accepting function parameters, so anonymous functions (lambda expressions) or named functions can be passed to them. The query methods with IEnumerable<T> output are pure functions. They are referential transparency and side effect free. When they are called, they only create and output a new sequence wrapping the input sequence and the query logic, with the query logic not executed, so there are no state changes, data mutation, I/O, etc. The query logic execution is deferred until the result values are pulled from the output sequence. The other query methods that output a new collection or a single value are impure functions. When they are called, they immediately execute the query and pull the results.

Sequence queries

As discussed in the Function composition and LINQ queries chapter, some query methods in this category can be written with query expressions syntax. In above query method list, these query methods are marked with *.

Generation

Enumerable type’s Empty, Range, Repeat methods can generate an IEnumerable<T> sequence. They are just normal static methods instead of extension methods:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TResult> Empty<TResult>();

public static IEnumerable<int> Range(int start, int count);

public static IEnumerable<TResult> Repeat<TResult>(TResult element, int count);

}

Empty just generates an IEnumerable<T> sequence, which contains no value:

internal static void Empty()

IEnumerable<string> empty = Enumerable.Empty<string>(); // Define query.

int count = 0;

foreach (string result in empty) // Execute query by pulling the results.

count++; // Not executed.

count.WriteLine(); // 0

}

Range generates an int sequence with the specified initial value and count of values:

internal static void Range()

IEnumerable<int>range = Enumerable.Range(-1, 5); // Define query.

foreach (int int32 in range) // Execute query.

int32.WriteLine();

}

To trace all values in a sequence, the following WriteLines extension method can be defined for IEnumerable<T>:

public static partial class TraceExtensions

public static void WriteLines<T>(this IEnumerable<T> values, Func<T, string> messageFactory = null)

if (messageFactory != null)

foreach (T value in values)

Trace.WriteLine(messageFactory(value));

else

foreach (T value in values)

Trace.WriteLine(value);

}

So, the above example can be simplified as:

internal static void Range()

IEnumerable<int> range = Enumerable.Range(-1, 5); // Define query.

range.WriteLines(); // Execute query. -1 0 1 2 3

}

The following example creates a sequence with large number of int values:

internal static void LargeRange()

IEnumerable<int>range = Enumerable.Range(1, int.MaxValue); // Define query.

}

As just mentioned, calling above LargeRange just defines a query. A large sequence is created, but each actual value in the large sequence is not generated.

internal static void Repeat()

IEnumerable<string>repeat = Enumerable.Repeat("*", 5); // Define query.

repeat.WriteLines(); // Execute query. * * * * *

}

DefaultIfEmpty generates a sequence based on the source sequence. If the source sequence is not empty, the returned sequence contains the same values from the source sequence. If the source sequence is empty, the returned sequence contains a single value, which is the default value of TSource type:

public static IEnumerable<TSource> DefaultIfEmpty<TSource>(this IEnumerable<TSource> source);

The other overload of DefaultIfEmpty allows to specify what default value to use if the source sequence is empty:

public static IEnumerable<TSource> DefaultIfEmpty<TSource>(

this IEnumerable<TSource> source, TSource defaultValue);

For example:

internal static void DefaultIfEmpty()

IEnumerable<int>source = Enumerable.Empty<int>();

IEnumerable<int>singletonIfEmpty = source.DefaultIfEmpty(); // Define query.

singletonIfEmpty.WriteLines(); // Execute query: 0

internal static void DefaultIfEmptyWithDefaultValue()

IEnumerable<int>source = Enumerable.Empty<int>();

IEnumerable<int>singletonIfEmpty = source.DefaultIfEmpty(1);

singletonIfEmpty.WriteLines(); // Execute query. 1

}

DefaultIfEmpty is also commonly used in left outer join, which is discussed later.

Filtering (restriction)

As demonstrated earlier, Where filters the values in the source sequence:

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> predicate);

Its predicate parameter is a callback function. When the query is executed, predicate is called with each value in the source sequence, and output a bool value. If output is true, this value is kept in the query result sequence; if output is false, this value is ignored. For example, the following query filters all types in .NET core library to get all primitive type:

private static readonly Assembly CoreLibrary = typeof(object).Assembly;

internal static void Where()

IEnumerable<Type> source = CoreLibrary.ExportedTypes;

IEnumerable<Type> primitives = source.Where(type => type.IsPrimitive); // Define query.

primitives.WriteLines(); // Execute query. System.Boolean System.Byte System.Char System.Double ...

}

And the equivalent query expression has a where clause:

internal static void Where()

IEnumerable<Type>source = CoreLibrary.ExportedTypes;

IEnumerable<Type>primitives = from type in source

where type.IsPrimitive

select type;

}

The other overload of Where accepts an indexed predicate function:

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source, Func<TSource, int, bool> predicate);

Here each time predicate is called with 2 parameters, a value in source sequence, and this value’s index in source sequence. For example:

internal static void WhereWithIndex()

IEnumerable<string>source = new string[] { "zero", "one", "two", "three", "four" };

IEnumerable<string>even = source.Where((value, index) => index % 2 == 0); // Define query.

even.WriteLines(); // Execute query. zero two four

}

The indexed Where overload is not supported in query expression syntax.

The other filtering query method is OfType. OfType is not supported in query expression either. It simply filters values by the specified type parameter:

internal static void OfType()

IEnumerable<object>source = new object[] { 1, 2, 'a', 'b', "aa", "bb", new object() };

IEnumerable<string>strings = source.OfType<string>(); // Define query.

strings.WriteLines(); // Execute query. aa bb

}

Mapping (projection)

Similar to Where, Select has 2 overloads, one accepts a selector function, the other one has an indexed selector function:

IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector);

IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, int, TResult> selector);

When the query is executed, the selector function is called with each TSource value, and map it to a TResult result in the output sequence. In the indexed overload, selector is also called with TSource value’s index. For example, the following Select query maps each integer to a formatted string that represents the integer’s square root:

internal static void Select()

IEnumerable<int>source = Enumerable.Range(0, 5);

IEnumerable<string>squareRoots = source.Select(int32 => $"{Math.Sqrt(int32):0.00}"); // Define query.

squareRoots.WriteLines(); // Execute query. 0.00 1.00 1.41 1.73 2.00

}

The equivalent query expression is a select clause with a single from clause:

internal static void Select()

IEnumerable<int>source = Enumerable.Range(0, 5);

IEnumerable<string>squareRoots = from int32 in source

select $"{Math.Sqrt(int32):0.00}";

}

Query expression must end with either a select clause, or group clause (discussed below). If there are other clauses between the starting from clause and the ending select clause, and the ending select clause simply has the value from the source sequence, then that ending select clause is ignored and is not compiled to a Select query method call. Above where query expression is such an example.

The following is an example of the indexed overload:

internal static IEnumerable<string> Words() => new string[] { "Zero", "one", "Two", "three", "four" };

internal static void SelectWithIndex()

IEnumerable<string>source = Words();

var mapped = source.Select((value, index) => new

Index = index,

Word = value.ToLowerInvariant()

}); // Define query: IEnumerable<(string Word, int Index)>

mapped.WriteLines(result => $"{result.Index}:{result.Word}"); // Execute query.

// 0:zero 1:one 2:two 3:three 4:four

}

Here selector returns anonymous type. As a result, Select returns a sequence of anonymous type, and var has to be used.

As discussed in the Immutability chapter, let clause is also compiled to Select query with a selector function returning anonymous type:

internal static void Let()

IEnumerable<int>source = Enumerable.Range(-2, 5);

IEnumerable<string>absoluteValues = from int32 in source

let abs = Math.Abs(int32)

where abs > 0

select $"Math.Abs({int32}) == {abs}";

}

The compiled Select query returns a (int int32, int abs) anonymous type:

internal static void CompiledLet()

IEnumerable<int>source = Enumerable.Range(-2, 5);

IEnumerable<string>absoluteValues = source

.Select(int32 => new { int32 = int32, abs = Math.Abs(int32) })

.Where(anonymous => anonymous.abs > 0)

.Select(anonymous => $"Math.Abs({anonymous.int32}):{anonymous.abs}"); // Define query.

absoluteValues.WriteLines(); // Execute query.

// Math.Abs(-2):2 Math.Abs(-1):1 Math.Abs(1):1 Math.Abs(2):2

}

SelectMany has 4 overloads. Similar to Where and Select, the following 2 overloads accept unindexed and indexed selector:

public static IEnumerable<TResult> SelectMany<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, IEnumerable<TResult>> selector);

public static IEnumerable<TResult> SelectMany<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, int, IEnumerable<TResult>>selector);

In contrast with Select, SelectMany’s selector is a one to many mapping. If there are N values from the source sequence, then they are mapped to N sequences. And eventually, SelectMany concatenates these N sequences into one single sequence. The following example calls SelectMany to query all members of all types in .NET core library, then filter the obsolete members (members with [Obsolete]):

internal static MemberInfo[] GetDeclaredMembers(this Type type) =>

type.GetMembers(

BindingFlags.Public | BindingFlags.Static | BindingFlags.Instance | BindingFlags.DeclaredOnly);

internal static bool IsObsolete(this MemberInfo member) =>

member.IsDefined(attributeType: typeof(ObsoleteAttribute), inherit: false);

internal static void SelectMany()

IEnumerable<Type>source = CoreLibrary.ExportedTypes;

IEnumerable<MemberInfo>oneToManyMapped = source.SelectMany(type => type.GetDeclaredMembers()); // Define query.

IEnumerable<MemberInfo> filtered = oneToManyMapped.Where(member => member.IsObsolete()); // Define query.

filtered.WriteLines(obsoleteMember => $"{obsoleteMember.DeclaringType}:{obsoleteMember}"); // Execute query.

// Equivalent to:

// foreach (MemberInfo obsoleteMember in filtered)

// {

// Trace.WriteLine($"{obsoleteMember.DeclaringType}:{obsoleteMember}");

// }

// ...

// System.Enum:System.String ToString(System.String, System.IFormatProvider)

// System.Enum:System.String ToString(System.IFormatProvider)

// ...

}

Apparently, the above SelectMany, Where, and are both extension methods for IEnumerable<T>, and they both return IEnumerable<T>, so that above LINQ query can be fluent, as expected:

internal static void FluentSelectMany()

IEnumerable<MemberInfo> mappedAndFiltered = CoreLibrary

.ExportedTypes

.SelectMany(type => type.GetDeclaredMembers())

.Where(member => member.IsObsolete()); // Define query.

mappedAndFiltered.WriteLines(obsoleteMember => $"{obsoleteMember.DeclaringType}:{obsoleteMember}"); // Execute query.

}

And the equivalent query expression has 2 from clauses:

internal static void SelectMany()

IEnumerable<MemberInfo>mappedAndFiltered =

from type in CoreLibrary.ExportedTypes

from member in type.GetPublicDeclaredMembers()

where member.IsObsolete()

select member;

}

Generally, SelectMany can flatten a hierarchical 2-level-sequence into a flat 1-level-sequence. In these examples, the source sequence is hierarchical – it has many types, and each type can have a sequence of many members. SelectMany flattens the hierarchy, and concatenates many sequences of members into a single sequence of members.

The other 2 SelectMany overloads accept 2 selector functions:

public static IEnumerable<TResult> SelectMany<TSource, TCollection, TResult>(

this IEnumerable<TSource> source, Func<TSource,

IEnumerable<TCollection>> collectionSelector,

Func<TSource, TCollection, TResult>resultSelector);

public static IEnumerable<TResult> SelectMany<TSource, TCollection, TResult>(

this IEnumerable<TSource> source,

Func<TSource, int, IEnumerable<TCollection>> collectionSelector,

Func<TSource, TCollection, TResult>resultSelector);

They accept 2 selector functions. The collection selector (non-indexed and index) maps source sequence’s each TSource value to many TCollection values (an IEnumerable<TCollection> sequence), and the result selector maps each TCollection value and its original TSource value to a TResult value. So eventually they still return a sequence of TResult values. For example, the following example use result selector to map type and member to string representation:

internal static void SelectManyWithResultSelector()

IEnumerable<Type> source = CoreLibrary.ExportedTypes;

IEnumerable<string> obsoleteMembers = source

.SelectMany(

collectionSelector: type => type.GetDeclaredMembers(),

resultSelector: (type, member) => new { Type = type, Member = member })

.Where(typeAndMember => typeAndMember.Member.IsObsolete())

.Select(typeAndMember => $"{typeAndMember.Type}:{typeAndMember.Member}");

}

The equivalent query expression has 2 from clauses for the SelectMany query, a where clause for Where, and 1 select query for Select:

internal static void SelectManyWithResultSelector()

IEnumerable<Type>source = CoreLibrary.ExportedTypes;

IEnumerable<string>obsoleteMembers =

from type in source

from member in type.GetDeclaredMembers()

where member.IsObsolete()

select $"{type}:{member}";

}

The collection selector function returns a sequence, which can be queried too. Here the Where query logically filters the obsolete member can be equivalently applied to the collection selector, which is called a subquery:

internal static void SelectManyWithResultSelectorAndSubquery()

IEnumerable<Type> source = CoreLibrary.ExportedTypes;

IEnumerable<string> obsoleteMembers = source.SelectMany(

collectionSelector: type => type.GetDeclaredMembers().Where(member => member.IsObsolete()),

resultSelector: (type, obsoleteMember) => $"{type}:{obsoleteMember}"); // Define query.

obsoleteMembers.WriteLines(); // Execute query.

}

The equivalent query expression has a sub query expression for Where:

internal static void SelectManyWithResultSelectorAndSubquery()

IEnumerable<Type>source = CoreLibrary.ExportedTypes;

IEnumerable<string>obsoleteMembers =

from type in source

from obsoleteMember in (from member in type.GetDeclaredMembers()

where member.IsObsolete()

select member)

select $"{type}:{obsoleteMember}"; // Define query.

obsoleteMembers.WriteLines(); // Execute query.

}

SelectMany is a very powerful query method, and the multiple from clauses is also a powerful syntax to build a functional workflow. This is discussed in the Category Theory chapters.

Grouping

The GroupBy method has 8 overloads. The minimum requirement is to specified a key selector function, which is called with each value in the source sequence, and return a key:

public static IEnumerable<IGrouping<TKey, TSource>> GroupBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

Each value from the source sequence is mapped to a key by calling the keys elector. If 2 keys are equal, these 2 source values are in the same group. Take the following persons as example:

internal class Person

internal Person(string name, string placeOfBirth)

this.Name = name;

this.PlaceOfBirth = placeOfBirth;

internal string Name { get; }

internal string PlaceOfBirth { get; }

internal static partial class QueryMethods

internal static IEnumerable<Person> Persons() => new Person[]

new Person(name: "Robert Downey Jr.", placeOfBirth: "US"),

new Person(name: "Tom Hiddleston", placeOfBirth: "UK"),

new Person(name: "Chris Hemsworth", placeOfBirth: "AU"),

new Person(name: "Chris Evans", placeOfBirth: "US"),

new Person(name: "Paul Bettany", placeOfBirth: "UK")

};

}

These Person instances represent actors of Marvel Cinematic Universe. They can be simply grouped by their place of birth:

internal static void GroupBy()

IEnumerable<Person>source = Persons();

IEnumerable<IGrouping<string, Person>>groups = source.GroupBy(person => person.PlaceOfBirth); // Define query.

foreach (IGrouping<string, Person> group in groups) // Execute query.

$"{group.Key}: ".Write();

foreach (Person person in group)

$"{person.Name}, ".Write();

Environment.NewLine.Write();

// US: Robert Downey Jr., Chris Evans,

// UK: Tom Hiddleston, Paul Bettany,

// AU: Chris Hemsworth,

}

GroupBy returns IEnumerable<IGrouping<TKey, TSource>>. The following is the definition of IGrouping<TKey, TElement> interface:

namespace System.Linq

public interface IGrouping<out TKey, out TElement> : IEnumerable<TElement>, IEnumerable

TKey Key { get; }

}

It is just an IEnumerable<T> sequence with an additional Key property. So, above GroupBy returns a hierarchical sequence. It is a sequence of groups, where each group is a sequence of values. The equivalent query expression is a group clause:

internal static void GroupBy()

IEnumerable<Person>source = Persons();

IEnumerable<IGrouping<string, Person>>groups = from person in source

group person by person.PlaceOfBirth;

}

GroupBy can also accepts a result selector function to map each group and its key to a result in the returned sequence:

public static IEnumerable<TResult> GroupBy<TSource, TKey, TResult>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector,

Func<TKey, IEnumerable<TSource>, TResult> resultSelector);

This overload, does not return of hierarchical sequence of groups, but flattened sequence of result values:

internal static void GroupByWithResultSelector()

IEnumerable<Person>source = Persons();

IEnumerable<string>groups = source

.GroupBy(

keySelector: person => person.PlaceOfBirth,

resultSelector: (key, group) => $"{key}:{group.Count()}"); // Define query.

groups.WriteLines(); // Execute query. US:2 UK:2 AU:1

}

This overload is directly not supported by query expression. However, its result selector can be equivalently applied with an additional Select query:

internal static void GroupByAndSelect()

IEnumerable<Person>source = Persons();

IEnumerable<IGrouping<string, Person>>groups = source.GroupBy(person => person.PlaceOfBirth);

IEnumerable<string>mapped = groups.Select(group => $"{group.Key}: {group.Count()}"); // Define query.

groups.WriteLines(); // Execute query. US:2 UK:2 AU:1

}

As just demonstrated, this GroupBy overload is equivalent to query expression with a group clause, and Select can be compiled from a select clause:

internal static void GroupByAndSelect()

IEnumerable<Person>source = Persons();

IEnumerable<IGrouping<string, Person>>groups = from person in source

group person by person.PlaceOfBirth;

IEnumerable<string>mapped = from @group in groups

select $"{@group.Key}: {@group.Count()}";

}

Here @ is prepended to the @group identifier, because group is a query keyword. By removing the groups variable, the first query expression becomes the second query expression’s subquery:

internal static void FluentGroupByAndSelect()

IEnumerable<Person>source = Persons();

IEnumerable<string>mapped = from @group in (from person in source

group person by person.PlaceOfBirth)

select $"{@group.Key}: {@group.Count()}";

}

The above expression is nested rather than fluent. So, the into query keyword is provided for continuation like this:

internal static void GroupByAndSelectWithInto()

IEnumerable<Person>source = Persons();

IEnumerable<string>mapped = from person in source

group person by person.PlaceOfBirth into @group

select $"{@group.Key}: {@group.Count()}";

}

The compilation of the above 2 query expressions are identical.

GroupBy can also accept an element selector function to map each value in the source sequence in the source sequence to a result value in the group:

public static IEnumerable<IGrouping<TKey, TElement>> GroupBy<TSource, TKey, TElement>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector,

Func<TSource, TElement> elementSelector);

For example:

internal static void GroupByWithElementSelector()

IEnumerable<Person>source = Persons();

IEnumerable<IGrouping<string, string>>groups = source

.GroupBy(

keySelector: person => person.PlaceOfBirth,

elementSelector: person => person.Name); // Define query.

foreach (IGrouping<string, string> group in groups) // Execute query.

$"{group.Key}: ".Write();

foreach (string name in group)

$"{name}, ".Write();

Environment.NewLine.Write();

// US: Robert Downey Jr., Chris Evans,

// UK: Tom Hiddleston, Paul Bettany,

// AU: Chris Hemsworth,

}

In query expression, the element selector can be specified after the group keyword:

internal static void GroupByWithElementSelector()

IEnumerable<Person>source = Persons();

IEnumerable<IGrouping<string, string>>groups = from person in source

group person.Name by person.PlaceOfBirth;

}

And element selector can be used with result selector:

public static IEnumerable<TResult> GroupBy<TSource, TKey, TElement, TResult>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector,

Func<TSource, TElement> elementSelector,

Func<TKey, IEnumerable<TElement>, TResult> resultSelector);

Again, result selector can flatten the hierarchical sequence:

internal static void GroupByWithElementAndResultSelector()

IEnumerable<Person>source = Persons();

IEnumerable<string>groups = source.GroupBy(

keySelector: person => person.PlaceOfBirth,

elementSelector: person => person.Name,

resultSelector: (key, group) => $"{key}: {string.Join(", ", group)}"); // Define query.

groups.WriteLines(); // Execute query.

// US: Robert Downey Jr., Chris Evans

// UK: Tom Hiddleston, Paul Bettany

// AU: Chris Hemsworth

}

Similar to SelectMany, GroupBy with both element selector and result selector is not directly supported in query expression. The result selector logic can be done with a select continuation:

internal static void GroupByWithElementSelectorAndSelect()

IEnumerable<Person>source = Persons();

IEnumerable<string>groups = from person in source

group person.Name by person.PlaceOfBirth into @group

select $"{@group.Key}: {string.Join(",", @group)}";

}

The rest 4 overloads accept an IEqualityComparer<TKey> interface:

public static IEnumerable<IGrouping<TKey, TSource>> GroupBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector, IEqualityComparer<TKey> comparer);

public static IEnumerable<TResult> GroupBy<TSource, TKey, TResult>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector,

Func<TKey, IEnumerable<TSource>, TResult> resultSelector,

IEqualityComparer<TKey>comparer);

public static IEnumerable<IGrouping<TKey, TElement>>GroupBy<TSource, TKey, TElement>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

IEqualityComparer<TKey>comparer);

public static IEnumerable<TResult> GroupBy<TSource, TKey, TElement, TResult>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

Func<TKey, IEnumerable<TElement>, TResult> resultSelector,

IEqualityComparer<TKey>comparer);

IEqualityComparer<TKey> provides the methods to determine whether 2 keys are equal when grouping all keys:

namespace System.Collections.Generic

public interface IEqualityComparer<in T>

bool Equals(T x, T y);

int GetHashCode(T obj);

}

For example:

internal static void GroupByWithEqualityComparer()

IEnumerable<Person>source = Persons();

IEnumerable<string>groups = source.GroupBy(

keySelector: person => person.PlaceOfBirth,

elementSelector: person => person.Name,

resultSelector: (key, group) => $"{key}:{string.Join(",", group)}",

comparer: StringComparer.OrdinalIgnoreCase); // Define query.

groups.WriteLines(); // Execute query. US:2 UK: 2 AU: 1

}

These 4 overloads are not supported by query expression.

Join

The most common join operations are inner join, left outer join and cross join. LINQ provides Join query method for inner join, and provides GroupJoin for left outer join. Cross join can be easily done with SelectMany or Join.

Inner join

Join is designed for inner join:

IEnumerable<TResult> Join<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer, IEnumerable<TInner> inner,

Func<TOuter, TKey> outerKeySelector, Func<TInner, TKey> innerKeySelector,

Func<TOuter, TInner, TResult> resultSelector)

IEnumerable<TResult> Join<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer, IEnumerable<TInner> inner,

Func<TOuter, TKey> outerKeySelector, Func<TInner, TKey> innerKeySelector,

Func<TOuter, TInner, TResult> resultSelector,

IEqualityComparer<TKey> comparer)

Each outer value from the outer source is mapped to an outer key by calling the outer key selector, and each inner value from the inner source is mapped to an inner key. When an outer key is equal to an inner key, the source outer value and the matching source inner value are paired, and mapped to a result by calling the result selector. So each outer value with a matching inner value is mapped to a result in the returned sequence, and each outer value without a matching inner value is ignored. Take the following characters as example:

internal partial class Character

internal Character(string name, string placeOfBirth, string starring)

this.Name = name;

this.PlaceOfBirth = placeOfBirth;

this.Starring = starring;

internal string Name { get; }

internal string PlaceOfBirth { get; }

internal string Starring { get; }

internal static partial class QueryMethods

internal static IEnumerable<Character> Characters() => new Character[]

new Character(name: "Tony Stark", placeOfBirth: "US", starring: "Robert Downey Jr."),

new Character(name: "Thor", placeOfBirth: "Asgard", starring: "Chris Hemsworth"),

new Character(name: "Steve Rogers", placeOfBirth: "US", starring: "Chris Evans"),

new Character(name: "Vision", placeOfBirth: "KR", starring: "Paul Bettany"),

new Character(name: "JARVIS", placeOfBirth: "US", starring: "Paul Bettany")

};

}

These Character instances represent characters in the movie Avengers 2, and can be joined with actors. When a character from outer sequence matches an actor from inner sequence by cast, these 2 values are paired and mapped to the result sequence:

internal static void InnerJoin()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

IEnumerable<string>innerJoin = outer.Join(

inner: inner,

outerKeySelector: person => person.Name,

innerKeySelector: character => character.Starring,

resultSelector: (person, character) => $"{person.Name} ({person.PlaceOfBirth}): {character.Name}"); // Define query.

innerJoin.WriteLines(); // Execute query.

// Robert Downey Jr. (US): Tony Stark

// Chris Hemsworth (AU): Thor

// Chris Evans (US): Steve Rogers

// Paul Bettany (UK): Vision

// Paul Bettany (UK): JARVIS

}

In the inner join results, the name “Tom Hiddleston” does not exist in the results, because the person with this name cannot match with any character’s starring (Tom Hiddleston is the actor of Loki, who is in Avengers 1 but not in Avengers 2). And the name “Paul Bettany” appears twice in the results, because the person with this name matches 2 characters’ starring (Paul Bettany is the voice of JARVIS and the actor of Vision). The equivalent query expression has a join clause:

internal static void InnerJoin()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

IEnumerable<string>innerJoin =

from person in outer

join character in inner on person.Name equals character.Starring

select $"{person.Name} ({person.PlaceOfBirth}): {character.Name}";

}

In the above example, the outer value and the inner value are matched with a single key - Person.Name property and Character.Starring property. To match with multiple keys, just have both outer key selector and inner key selector return the same anonymous type with multiple properties:

internal static void InnerJoinWithMultipleKeys()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

IEnumerable<string>innerJoin = outer.Join(

inner: inner,

outerKeySelector: person => new { Starring = person.Name, PlaceOfBirth = person.PlaceOfBirth },

innerKeySelector: character => new { Starring = character.Starring, PlaceOfBirth = character.PlaceOfBirth },

resultSelector: (person, character) =>

$"{person.Name} ({person.PlaceOfBirth}): {character.Name} ({character.PlaceOfBirth})"); // Define query.

innerJoin.WriteLines(); // Execute query.

// Robert Downey Jr. (US): Tony Stark (US)

// Chris Evans (US): Steve Rogers (US)

}

Anonymous type can be also used with join clause in query expression:

internal static void InnerJoinWithMultiKeys()

IEnumerable<Person>outer = Persons();

IEnumerable<Character> inner = Characters();

IEnumerable<string>innerJoin =

from person in outer

join character in inner

on new { Starring = person.Name, PlaceOfBirth = person.PlaceOfBirth }

equals new { Starring = character.Starring, PlaceOfBirth = character.PlaceOfBirth }

select $"{person.Name} ({person.PlaceOfBirth}): {character.Name} ({character.PlaceOfBirth})";

}

Left outer join

GroupJoin is designed for left outer join:

IEnumerable<TResult> GroupJoin<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer, IEnumerable<TInner> inner,

Func<TOuter, TKey> outerKeySelector, Func<TInner, TKey> innerKeySelector,

Func<TOuter, IEnumerable<TInner>, TResult> resultSelector)

IEnumerable<TResult> GroupJoin<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer, IEnumerable<TInner> inner,

Func<TOuter, TKey> outerKeySelector, Func<TInner, TKey> innerKeySelector,

Func<TOuter, IEnumerable<TInner>, TResult> resultSelector,

IEqualityComparer<TKey> comparer)

Each outer value from the outer source is mapped to an outer key by calling the outer key selector, and each inner value from the inner source is mapped to an inner key. When an outer key is equal to zero, one, or more inner key, the source outer value and all the matching source inner values are paired, and mapped to a result by calling the result selector. So each outer value with or without matching inner values is mapped to a result in the returned sequence. It is called GroupJoin, because each outer value is paired with a group of matching inner values. If there is no matching inner value, the outer value is paired with an empty group:

internal static void LeftOuterJoin()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

var leftOuterJoin = outer.GroupJoin(

inner: inner,

outerKeySelector: person => person.Name,

innerKeySelector: character => character.Starring,

resultSelector: (person, charactersGroup) =>

new { Person = person, Characters = charactersGroup }); // Define query.

foreach (var result in leftOuterJoin) // Execute query.

$"{result.Person.Name} ({result.Person.PlaceOfBirth}): ".Write();

foreach (Character character in result.Characters)

$"{character.Name} ({character.PlaceOfBirth}), ".Write();

Environment.NewLine.Write();

// Robert Downey Jr. (US): Tony Stark (US),

// Tom Hiddleston (UK):

// Chris Hemsworth (AU): Thor (Asgard),

// Chris Evans (US): Steve Rogers (US),

// Paul Bettany (UK): Vision (KR), JARVIS (US),

}

Here result selector is called with each actor, and a group of matching characters, then it returns anonymous type consists of both the actor and the matching characters. So eventually GroupJoin returns a hierarchical sequence. In the results, the person with name “Tom Hiddleston” matches no character, so it is paired with an empty Character group, and each other person matches 1 or more characters, so is paired with a non-empty Character group. In query expression, GroupJoin is equivalent to join clause with the into keyword:

internal static void LeftOuterJoin()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

var leftOuterJoin =

from person in outer

join character in inner on person.Name equals character.Starring into charactersGroup

select new { Person = person, Characters = charactersGroup };

}

In the join clause, into does not mean a continuation. it is a part of the join.

The hierarchical sequence returned by GroupJoin can be flattened by SelectMany. In this kind of flattening scenario, DefaultIfEmpty is usually used:

internal static void LeftOuterJoinWithDefaultIfEmpty()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

var leftOuterJoin = outer

.GroupJoin(

inner: inner,

outerKeySelector: person => person.Name,

innerKeySelector: character => character.Starring,

resultSelector: (person, charactersGroup) => new { Person = person, Characters = charactersGroup })

.SelectMany(

collectionSelector: group => group.Characters.DefaultIfEmpty(),

resultSelector: (group, character) => new { Person = group.Person, Character = character }); // Define query.

leftOuterJoin.WriteLines(result => $"{result.Person.Name}: {result.Character?.Name}");

// Robert Downey Jr.: Tony Stark

// Tom Hiddleston:

// Chris Hemsworth: Thor

// Chris Evans: Steve Rogers

// Paul Bettany: Vision

// Paul Bettany: JARVIS

}

Without the DefaultIfEmpty call, the second result “Tom Hiddleston” is ignored in the result sequence. The equivalent query expression has 2 from clauses for SelectMany:

internal static void LeftOuterJoinWithDefaultIfEmpty()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

var leftOuterJoin =

from person in outer

join character in inner on person.Name equals character.Starring into charactersGroup

from character in charactersGroup.DefaultIfEmpty()

select new { Person = person, Character = character };

}

There is already a from clause before join clause, so, just add one more from clause after join clause.

Left outer join can also implement by mapping each outer value with all filtered matching inner values:

internal static void LeftOuterJoinWithSelect()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

var leftOuterJoin = outer.Select(person => new

Person = person,

Characters = inner.Where(character =>

EqualityComparer<string>.Default.Equals(person.Name, character.Starring))

}); // Define query.

foreach (var result in leftOuterJoin) // Execute query.

$"{result.Person.Name} ({result.Person.PlaceOfBirth}): ".Write();

foreach (Character character in result.Characters)

$"{character.Name} ({character.PlaceOfBirth}), ".Write();

Environment.NewLine.Write();

// Robert Downey Jr. (US): Tony Stark (US),

// Tom Hiddleston (UK):

// Chris Hemsworth (AU): Thor (Asgard),

// Chris Evans (US): Steve Rogers (US),

// Paul Bettany (UK): Vision (KR), JARVIS (US),

}

Notice here the Where subquery filters all inner values for each outer value. Generally, left outer join can be implemented with mapping query and filtering subquery:

internal static IEnumerable<TResult> LeftOuterJoinWithSelect<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, IEnumerable<TInner>, TResult> resultSelector,

IEqualityComparer<TKey>comparer = null)

comparer = comparer ?? EqualityComparer<TKey>.Default;

return outer.Select(outerValue => resultSelector(

outerValue,

inner.Where(innerValue => comparer.Equals(outerKeySelector(outerValue), innerKeySelector(innerValue)))));

}

In query expression, it just a simple query expression with a select clause containing a subquery with a where clause:

internal static void LeftOuterJoinWithSelect()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

var leftOuterJoin =

from person in outer

select new

Person = person,

Characters = from character in inner

where EqualityComparer<string>.Default.Equals(person.Name, character.Starring)

select character

};

internal static IEnumerable<TResult> LeftOuterJoinWithSelect<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, IEnumerable<TInner>, TResult> resultSelector,

IEqualityComparer<TKey>comparer = null)

comparer = comparer ?? EqualityComparer<TKey>.Default;

return from outerValue in outer

select resultSelector(

outerValue,

(from innerValue in inner

where comparer.Equals(outerKeySelector(outerValue), innerKeySelector(innerValue))

select innerValue));

}

The difference is, for N outer values, GroupJoin pull all inner values once and cache them, Select and Where does not cache anything and pull all inner values N times. The internal implementation of these query methods are discussed later in this chapter.

Cross Join

Cross join 2 sequences is to return the Cartesian product of values in those 2 sequences. The easiest way for cross join is SelectMany:

private static readonly int[] Rows = { 1, 2, 3 };

private static readonly string[] Columns = { "A", "B", "C", "D" };

internal static void CrossJoin()

IEnumerable<string>cells = Rows

.SelectMany(row => Columns, (row, column) => $"{column}{row}"); // Define query.

int cellIndex = 0;

int columnCount = Columns.Length;

foreach (string cell in cells) // Execute query.

$"{cell} ".Write();

if (++cellIndex % columnCount == 0)

Environment.NewLine.Write();

// A1 B1 C1 D1

// A2 B2 C2 D2

// A3 B3 C3 D3

}

Notice here all inner values are pulled for each outer value. If outer sequence has N outer values, then the inner sequence are iterated N times. In query expression, as fore mentioned, 2 from clauses are compiled to SelectMany:

internal static void CrossJoin()

IEnumerable<string>cells = from row in Rows

from column in Columns

select $"{column}{row}";

}

A general CrossJoin query method can be implemented as:

internal static IEnumerable<TResult> CrossJoin<TOuter, TInner, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TInner, TResult>resultSelector) =>

outer.SelectMany(outerValue => inner, resultSelector);

// Equivalent to:

// from outerValue in outer

// from innerValue in inner

// select resultSelector(outerValue, innerValue);

Cross join can also be done with Join, with inner key always equal to outer key, so that each outer value matches all inner values:

internal static void CrossJoinWithJoin()

IEnumerable<string>cells = Rows.Join(

inner: Columns,

outerKeySelector: row => true,

innerKeySelector: column => true,

resultSelector: (row, column) => $"{column}{row}"); // Define query.

int cellIndex = 0;

int columnCount = Columns.Length;

foreach (string cell in cells) // Execute query.

$"{cell} ".Write();

if (++cellIndex % columnCount == 0)

Environment.NewLine.Write();

}

And generally, cross join can be implemented by Join as:

internal static IEnumerable<TResult> CrossJoinWithJoin<TOuter, TInner, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TInner, TResult>resultSelector) =>

outer.Join(

inner: inner,

outerKeySelector: outerValue => true,

innerKeySelector: innerValue => true,

resultSelector: resultSelector); // Equivalent to:

// Equivalent to:

// from outerValue in outer

// join innerValue in inner on true equals true

// select resultSelector(outerValue, innerValue);

In query expression, again, Join is just a join clause without into:

internal static void CrossJoinWithJoin()

IEnumerable<string>cells = from row in Rows

join column in Columns on true equals true

select $"{column}{row}";

internal static IEnumerable<TResult> CrossJoinWithJoin<TOuter, TInner, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TInner, TResult>resultSelector) =>

from outerValue in outer

join innerValue in inner on true equals true

select resultSelector(outerValue, innerValue);

The above inner join can be logically viewed as cross join with filtering the matching outer value and inner value. The above inner join of persons and characters can be implemented with SelectMany and Where as:

internal static void InnerJoinWithSelectMany()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

IEnumerable<string>innerJoin = outer

.SelectMany(

collectionSelector: person => inner,

resultSelector: (person, character) => new { Person = person, Character = character })

.Where(crossJoinValue => EqualityComparer<string>.Default.Equals(

crossJoinValue.Person.Name, crossJoinValue.Character.Starring))

.Select(innerJoinValue =>

$"{innerJoinValue.Person.Name} ({innerJoinValue.Person.PlaceOfBirth}): {innerJoinValue.Character.Name}");

// Define query.

innerJoin.WriteLines(); // Execute query.

// Robert Downey Jr. (US): Tony Stark

// Chris Hemsworth (AU): Thor

// Chris Evans (US): Steve Rogers

// Paul Bettany (UK): Vision

// Paul Bettany (UK): JARVIS

}

Generally, inner join and be implemented with cross join and filtering:

internal static IEnumerable<TResult> InnerJoinWithSelectMany<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, TInner, TResult>resultSelector,

IEqualityComparer<TKey>comparer = null)

comparer = comparer ?? EqualityComparer<TKey>.Default;

return outer

.SelectMany(

collectionSelector: outerValue => inner,

resultSelector: (outerValue, innerValue) => new { OuterValue = outerValue, InnerValue = innerValue })

.Where(

crossJoinValue => comparer.Equals(

outerKeySelector(crossJoinValue.OuterValue),

innerKeySelector(crossJoinValue.InnerValue)))

.Select(innerJoinValue => resultSelector(innerJoinValue.OuterValue, innerJoinValue.InnerValue));

}

In query expression, as fore mentioned, SelectMany is 2 from clauses:

internal static void InnerJoinWithSelectMany()

IEnumerable<Person>outer = Persons();

IEnumerable<Character>inner = Characters();

IEnumerable<string>innerJoin =

from person in outer

from character in inner

where EqualityComparer<string>.Default.Equals(person.Name, character.Starring)

select $"{person.Name} ({person.PlaceOfBirth}): {character.Name}";

internal static IEnumerable<TResult> InnerJoinWithSelectMany<TOuter, TInner, TKey, TResult>(

this IEnumerable<TOuter> outer,

IEnumerable<TInner>inner,

Func<TOuter, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<TOuter, TInner, TResult>resultSelector,

IEqualityComparer<TKey>comparer = null)

comparer = comparer ?? EqualityComparer<TKey>.Default;

return from outerValue in outer,

from innerValue in inner

where comparer.Equals(outerKeySelector(outerValue), innerKeySelector(innerValue))

select resultSelector(outerValue, innerValue);

}

The difference is, for N outer values, Join pull all inner values once and cache them, SelectMany does not cache anything and pull all inner values N times. Again, the internal implementation of these query methods is discussed later in this chapter.

Concatenation

Concat merges 2 sequences by putting the second sequence’s values after the first sequence’s values:

public static IEnumerable<TSource> Concat<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second);

For example:

internal static int[] First() => new int[] { 1, 2, 3, 4, 4 };

internal static int[] Second() => new int[] { 3, 4, 5, 6 };

internal static void Concat()

IEnumerable<int>first = First();

IEnumerable<int>second = Second();

IEnumerable<int>concat = first.Concat(second); // Define query.

concat.WriteLines(); // Execute query. 1 2 3 4 4 3 4 5 6

}

.NET Core provides Prepend/Append, which merge the specified value to the beginning/end of the source sequence:

public static IEnumerable<TSource> Prepend<TSource>(this IEnumerable<TSource> source, TSource element);

public static IEnumerable<TSource> Append<TSource>(this IEnumerable<TSource> source, TSource element);

For example:

internal static void AppendPrepend()

IEnumerable<int>prepend = Enumerable.Range(0, 5).Prepend(-1); // Define query.

prepend.WriteLines(); // Execute query. -1 0 1 2 3 4

IEnumerable<int> append = Enumerable.Range(0, 5).Append(-1); // Define query.

append.WriteLines(); // Execute query. 0 1 2 3 4 -1

}

Set

Distinct accepts a source sequence, and returns a set, where duplicate values are removed:

public static IEnumerable<TSource> Distinct<TSource>(this IEnumerable<TSource> source);

For example:

internal static void Distinct()

IEnumerable<int>first = First();

IEnumerable<int>distinct = first.Distinct(); // Define query.

distinct.WriteLines(); // Execute query. 1 2 3 4

}

The following query methods accepts 2 sequences and returns a set:

public static IEnumerable<TSource> Union<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second);

public static IEnumerable<TSource> Intersect<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second);

public static IEnumerable<TSource> Except<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second);

In contrast to Concat, Union adds 2 sequences as if they are sets, and returns their set union, which is equivalent to concatenating 2 sequences with duplicate values removed:

internal static void Union()

IEnumerable<int>first = First();

IEnumerable<int>second = Second();

IEnumerable<int>union = first.Union(second); // Define query.

union.WriteLines(); // Execute query. 1 2 3 4 5 6

}

Intersect returns 2 sequences’ set intersection, the distinct values that 2 sequences have in common:

internal static void Intersect()

IEnumerable<int>first = First();

IEnumerable<int>second = Second();

IEnumerable<int>intersect = first.Intersect(second); // Define query.

intersect.WriteLines(); // Execute query. 3 4

}

Except returns the set complement of 2 sequences, by subtracting the second sequence from the first one:

internal static void Except()

IEnumerable<int>first = First();

IEnumerable<int>second = Second();

IEnumerable<int>except = first.Except(second); // Define query.

except.WriteLines(); // Execute query. 1 2

}

There are other overloads that accepts a comparer:

public static IEnumerable<TSource> Distinct<TSource>(

this IEnumerable<TSource> source, IEqualityComparer<TSource> comparer);

public static IEnumerable<TSource> Union<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second, IEqualityComparer<TSource> comparer);

public static IEnumerable<TSource> Intersect<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second, IEqualityComparer<TSource> comparer);

public static IEnumerable<TSource> Except<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second, IEqualityComparer<TSource> comparer);

For example:

internal static void DistinctWithComparer()

IEnumerable<string>source = new string[] { "aa", "AA", "Aa", "aA", "bb" };

IEnumerable<string>distinctWithComparer = source.Distinct(StringComparer.OrdinalIgnoreCase); // Define query.

distinctWithComparer.WriteLines(); // Execute query. aa bb

}

Convolution

Zip is provided since .NET Framework 4.0. It accepts 2 sequences and returns their convolution:

public static IEnumerable<TResult> Zip<TFirst, TSecond, TResult>(

this IEnumerable<TFirst> first, IEnumerable<TSecond> second, Func<TFirst, TSecond, TResult> resultSelector);

It calls result selector to map 2 values (each value from each sequence) to a result in the returned sequence:

internal static void Zip()

IEnumerable<int>first = First();

IEnumerable<int>second = Second();

IEnumerable<int>zip = first.Zip(second, (a, b) => a + b); // Define query.

zip.WriteLines(); // Execute query. 4 6 8 10

}

When one input sequence has more values than the other, those values are ignored. Here the first sequence { 1, 2, 3, 4, 4 } and second sequence { 3, 4, 5, 6 } are zipped to a new sequence { 1 + 3, 2 + 4, 3 + 5, 4 + 6 }. The first sequence has one more value than the second, so its last value 4 is ignored.

Partitioning

Partitioning query methods are straightforward. Skip/Take simply skips/takes the specified number of values in the source sequence:

public static IEnumerable<TSource> Skip<TSource>(this IEnumerable<TSource> source, int count);

public static IEnumerable<TSource> Take<TSource>(this IEnumerable<TSource> source, int count);

For example:

internal static void SkipTake()

IEnumerable<int>source = Enumerable.Range(0, 5);

IEnumerable<int> partition1 = source.Skip(2); // Define query.

partition1.WriteLines(); // Execute query. 2 3 4

IEnumerable<int> partition2 = source.Take(2); // Define query.

partition2.WriteLines(); // Execute query. 0 1

}

SkipWhile/TakeWhile accept a predicate function:

public static IEnumerable<TSource> SkipWhile<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate);

public static IEnumerable<TSource> TakeWhile<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool>predicate);

SkipWhile/TakeWhile skips/takes values while predicate is called with each value and returns true. Once predicate is called with a value and returns false, SkipWhile/TakeWhile stop partitioning:

internal static void TakeWhileSkipWhile()

IEnumerable<int>source = new int[] { 1, 2, 3, -1, 4, 5 };

IEnumerable<int>partition1 = source.TakeWhile(int32 => int32 > 0); // Define query.

partition1.WriteLines(); // Execute query. 1 2 3

IEnumerable<int> partition2 = source.SkipWhile(int32 => int32 > 0); // Define query.

partition2.WriteLines(); // Execute query. -1 4 5

}

Just like Where and Select, SkipWhile/TakeWhile also have the indexed overload:

public static IEnumerable<TSource> SkipWhile<TSource>(

this IEnumerable<TSource> source, Func<TSource, int, bool> predicate);

public static IEnumerable<TSource> TakeWhile<TSource>(

this IEnumerable<TSource> source, Func<TSource, int, bool> predicate);

For example:

internal static void TakeWhileSkipWhileWithIndex()

IEnumerable<int>source = new int[] { 4, 3, 2, 1, 5 };

IEnumerable<int>partition1 = source.TakeWhile((int32, index) => int32 >= index); // Define query.

partition1.WriteLines(); // Execute query. 4 3 2

IEnumerable<int> partition2 = source.SkipWhile((int32, index) => int32 >= index); // Define query.

partition2.WriteLines(); // Execute query. 1 5

}

Ordering

The ordering methods are OrderBy and OrderByDescending:

IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector)

IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector, IComparer<TKey> comparer)

IOrderedEnumerable<TSource> OrderByDescending<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector)

IOrderedEnumerable<TSource> OrderByDescending<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector, IComparer<TKey> comparer)

The key selector specifies what should be compared to determine the order of values in the result sequence:

internal static void OrderBy()

IEnumerable<string>source = Words();

IEnumerable<string>ordered = source.OrderBy(word => word); // Define query.

ordered.WriteLines(); // Execute query. four one three Two Zero

source.WriteLines(); // Original sequence. Zero one Two three four

internal static void OrderByDescending()

IEnumerable<string>source = Words();

IEnumerable<string>ordered = source.OrderByDescending(word => word); // Define query.

ordered.WriteLines(); // Execute query. Zero Two three one four

source.WriteLines(); // Original sequence. Zero one Two three four

}

Here each value from the source sequence uses itself as the key for ordering. Also, as demonstrated above, OrderBy returns a new sequence, so OrderBy/OrderByDescending does not impact the source sequence. The equivalent query expression has an orderby clause:

internal static void OrderBy()

IEnumerable<string>source = Words();

IEnumerable<string>ordered = from word in source

orderby word ascending // ascending can be omitted.

select word;

internal static void OrderByDescending()

IEnumerable<string>source = Words();

IEnumerable<string>ordered = from word in source

orderby word descending

select word;

}

The comparer can be specified to provides the method to compare 2 keys:

namespace System.Collections.Generic

public interface IComparer<in T>

int Compare(T x, T y);

}

Compare returns an integer to determine the 2 values’ relative position in the ordered sequence. If x is less than y, Compare returns negative int value; If x is equal to y, Compare returns 0; If x is greater than y, Compare returns positive int value. For example:

internal static void OrderByWithComparer()

IEnumerable<string>source = Words();

IEnumerable<string>ordered = source.OrderBy(

keySelector: word => word, comparer: StringComparer.Ordinal); // Define query.

ordered.WriteLines(); // Execute query. Two Zero four one three

}

Here StringComparer.Ordinal provides a case-sensitive comparison. “Zero” comes to the first position of the result sequence, because upper case letter is less than lower case letter. This overload with comparer is not supported in query expression. When using the other overload without comparer, OrderBy/OrderByDescending uses System.Collections.Generic.Comparer<TKey>.Default. In the first OrderBy example, Comparer<string>.Default is used, which is equivalent to StringComparer.CurrentCulture.

As fore mentioned, ThenBy/ThenByDescending are extension methods of IOrderedEnumerable<T>, not IEnumerable<T>:

IOrderedEnumerable<TSource> ThenBy<TSource, TKey>(

this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector)

IOrderedEnumerable<TSource> ThenBy<TSource, TKey>(

this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector, IComparer<TKey> comparer)

IOrderedEnumerable<TSource> ThenByDescending<TSource, TKey>(

this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector)

IOrderedEnumerable<TSource> ThenByDescending<TSource, TKey>(

this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector, IComparer<TKey> comparer)

So they can be composed right after OrderBy/OrderByDescending:

internal static void ThenBy()

IEnumerable<Person>source = Persons();

IEnumerable<Person>ordered = source // IEnumerable<Person>

.OrderBy(person => person.PlaceOfBirth) // IOrderedEnumerable<Person>

.ThenBy(person => person.Name); // IOrderedEnumerable<Person>

ordered.WriteLines(person => $"{person.PlaceOfBirth}: {person.Name}"); // Execute query.

// AU: Chris Hemsworth

// UK: Paul Bettany

// UK: Tom Hiddleston

// US: Chris Evans

// US: Robert Downey Jr.

}

In the above example, persons are ordered by place of birth. If there are Person objects with the same PlaceOfBirth, they are ordered by Name. The query expression can have multiple key selectors in the orderby clause:

internal static void ThenBy()

IEnumerable<Person>source = Persons();

IEnumerable<Person>ordered = from person in source

orderby person.PlaceOfBirth, person.Name

select person;

}

Notice OrderBy can also be called after calling OrderBy:

internal static void OrderByAndOrderBy()

IEnumerable<Person>source = Persons();

IEnumerable<Person>ordered = source

.OrderBy(person => person.PlaceOfBirth)

.OrderBy(person => person.Name); // Define query.

ordered.WriteLines(person => $"{person.PlaceOfBirth}: {person.Name}"); // Execute query.

// US: Chris Evans

// AU: Chris Hemsworth

// UK: Paul Bettany

// US: Robert Downey Jr.

// UK: Tom Hiddleston

}

OrderBy with OrderBy is totally different from OrderBy with ThenBy. Here persons are ordered by place of birth. Then, all persons are ordered again by name. The equivalent query expression is:

internal static void OrderByOrderBy1()

IEnumerable<Person>source = Persons();

IEnumerable<Person>ordered = from person in source

orderby person.PlaceOfBirth

orderby person.Name

select person;

}

To makes it more intuitive, it can be separated to 2 query expressions:

internal static void OrderByOrderBy2()

IEnumerable<Person>source = Persons();

IEnumerable<Person>ordered1 = from person in source

orderby person.PlaceOfBirth

select person;

IEnumerable<Person>ordered2 = from person in ordered1

orderby person.Name

select person;

}

Apparently, both orderby clauses work on the entire input sequence. As fore mentioned, the into query keyword is for this kind scenario of continuation:

internal static void OrderByOrderBy3()

IEnumerable<Person>source = Persons();

IEnumerable<Person>ordered = from person in source

orderby person.PlaceOfBirth

select person into person

orderby person.Name

select person;

}

The compilation of the above 3 queries are identical.

Reverse simply reverses the positions of values:

public static IEnumerable<TSource> Reverse<TSource>(this IEnumerable<TSource> source)

For example:

internal static void Reverse()

IEnumerable<int>source = Enumerable.Range(0, 5);

IEnumerable<int>reversed = source.Reverse(); // Define query.

reversed.WriteLines(); // Execute query. 4 3 2 1 0

}

Conversion

Cast converts each value in source sequence to the specified type:

public static IEnumerable<TResult> Cast<TResult>(this IEnumerable source);

Unlike other query methods, Cast is an extension method of non-generic sequence, so it can work with types implementing either IEnumerable or IEnumerable<T>. So it can enable LINQ query for legacy types. The following example calls Microsoft Team Foundation Service (TFS) client APIs to query work items, where Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemCollection is returned. WorkItemCollection is a collection of Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItem, but it only implements IEnumerable, so it can be cast to a generic IEnumerable<WorkItem> safely, and further LINQ query can be applied. The following example execute a WIQL (Work Item Query Language of TFS) statement to query work items from TFS. Since WIQL does not support GROUP BY clause, the work items can be grouped locally with LINQ:

#if NETFX

internal static void CastNonGeneric(VssCredentials credentials)

using (TfsTeamProjectCollection projectCollection = new TfsTeamProjectCollection(

new Uri("https://dixin.visualstudio.com/DefaultCollection"), credentials))

// WorkItemCollection implements IEnumerable.

const string Wiql = "SELECT * FROM WorkItems WHERE [Work Item Type] = 'Bug' AND State != 'Closed'"; // WIQL does not support GROUP BY.

WorkItemStore workItemStore = (WorkItemStore)projectCollection.GetService(typeof(WorkItemStore));

WorkItemCollection workItems = workItemStore.Query(Wiql);

IEnumerable<WorkItem>genericWorkItems = workItems.Cast<WorkItem>(); // Define query.

IEnumerable<IGrouping<string, WorkItem>> workItemGroups = genericWorkItems

.GroupBy(workItem => workItem.CreatedBy); // Group work items locally.

// ...

#endif

The other non-generic sequences, like System.Resources.ResourceSet, System.Resources.ResourceReader, can be cast in the same way:

internal static void CastMoreNonGeneric()

// ResourceSet implements IEnumerable.

ResourceSet resourceSet = new ResourceManager(typeof(Resources))

.GetResourceSet(CultureInfo.CurrentCulture, createIfNotExists: true, tryParents: true);

IEnumerable<DictionaryEntry>entries1 = resourceSet.Cast<DictionaryEntry>();

// ResourceReader implements IEnumerable.

Assembly assembly = typeof(QueryMethods).Assembly;

using (Stream stream = assembly.GetManifestResourceStream(assembly.GetManifestResourceNames()[0]))

using (ResourceReader resourceReader = new ResourceReader(stream))

IEnumerable<DictionaryEntry>entries2 = resourceReader.Cast<DictionaryEntry>();

}

In query expression syntax, just specify the type in from clause before the value name:

#if NETFX

internal static void CastNonGeneric(VssCredentials credentials)

// WorkItemCollection implements IEnumerable.

using (TfsTeamProjectCollection projectCollection = new TfsTeamProjectCollection(

new Uri("https://dixin.visualstudio.com/DefaultCollection"), credentials))

const string Wiql = "SELECT * FROM WorkItems WHERE [Work Item Type] = 'Bug' AND State != 'Closed'"; // WIQL does not support GROUP BY.

WorkItemStore workItemStore = (WorkItemStore)projectCollection.GetService(typeof(WorkItemStore));

WorkItemCollection workItems = workItemStore.Query(Wiql);

IEnumerable<IGrouping<string, WorkItem>>workItemGroups =

from WorkItem workItem in workItems // Cast.

group workItem by workItem.CreatedBy; // Group work items in local memory.

// ...

#endif

internal static void CastMoreNonGenericI()

// ResourceSet implements IEnumerable.

ResourceSet resourceSet = new ResourceManager(typeof(Resources))

.GetResourceSet(CultureInfo.CurrentCulture, createIfNotExists: true, tryParents: true);

IEnumerable<DictionaryEntry>entries1 =

from DictionaryEntry entry in resourceSet // Cast.

select entry;

// ResourceReader implements IEnumerable.

Assembly assembly = typeof(QueryMethods).Assembly;

using (Stream stream = assembly.GetManifestResourceStream(assembly.GetManifestResourceNames()[0]))

using (ResourceReader resourceReader = new ResourceReader(stream))

IEnumerable<DictionaryEntry>entries2 =

from DictionaryEntry entry in resourceReader // Cast.

select entry;

}

And of course Cast can be used to generic IEnumerable<T>:

internal static void CastGenericIEnumerable()

IEnumerable<Base>source = new Base[] { new Derived(), new Derived() };

IEnumerable<Derived>cast = source.Cast<Derived>(); // Define query.

cast.WriteLines(result => result.GetType().Name); // Execute query. Derived Derived

}

And the query expression syntax is the same:

internal static void CastGenericIEnumerable()

IEnumerable<Base>source = new Base[] { new Derived(), new Derived() };

IEnumerable<Derived>cast = from Derived derived in source

select derived;

}

Cast must be used with caution, because type conversion can fail at runtime, for example:

internal static void CastGenericIEnumerableWithException()

IEnumerable<Base>source = new Base[] { new Derived(), new Base() };

IEnumerable<Derived>cast = source.Cast<Derived>(); // Define query.

cast.WriteLines(result => result.GetType().Name); // Execute query. Derived InvalidCastException

}

An InvalidCastException is thrown because the second value is of Base type, and cannot be cast to Derived.

The same query expression cast syntax can also be used in join clause:

internal static void CastWithJoin()

IEnumerable outer = new int[] { 1, 2, 3 };

IEnumerable inner = new string[] { "a", "bb", "ccc" };

IEnumerable<string>innerJoin = from int int32 in outer

join string @string in inner on int32 equals @string.Length

select $"{int32}: {@string}";

}

It is compiled to:

internal static void CastWithJoin()

IEnumerable outer = new int[] { 1, 2, 3 };

IEnumerable inner = new string[] { string.Empty, "a", "bb", "ccc", "dddd" };

IEnumerable<string>innerJoin = outer.Cast<int>().Join(

inner: inner.Cast<string>(),

outerKeySelector: int32 => int32,

innerKeySelector: @string => @string.Length, // on int32 equal @string.Length

resultSelector: (int32, @string) => $"{int32}:{@string}"); // Define query.

innerJoin.WriteLines(); // Execute query. 1:a 2:bb 3:ccc

}

Cast looks similar to the fore mentioned OfType method, which also can have the result type specified. However, they are very different, OfType filters the values of the specified type. If there are values not of the specified type, they are simply ignored. There no conversion so there is no chance of InvalidCastException.

AsEnumerable is a query method doing nothing. It accepts a source sequence, then return the source sequence itself:

public static IEnumerable<TSource> AsEnumerable<TSource>(this IEnumerable<TSource> source);

Its purpose is to make more derived type be visible only as IEnumerable<T>, and hide additional members of that more derived type:

internal static void AsEnumerable()

List<int>list = new List<int>();

list.Add(0);

IEnumerable<int>sequence = list.AsEnumerable(); // Add method is no longer available.

}

If the more derived source has method with the same signature as IEnumerable<T>’s extension method, after calling AsEnumerable, that IEnumerable<T> extension method is called:

internal static void AsEnumerableReverse()

List<int>list = new List<int>();

list.Reverse(); // List<T>.Reverse.

list

.AsEnumerable() // IEnumerable<T>.

.Reverse(); // Enumerable.Reverse.

SortedSet<int> sortedSet = new SortedSet<int>();

sortedSet.Reverse(); // SortedSet<T>.Reverse.

sortedSet.AsEnumerable().Reverse(); // Enumerable.Reverse.

ReadOnlyCollectionBuilder<int> readOnlyCollection = new ReadOnlyCollectionBuilder<int>();

readOnlyCollection.Reverse(); // ReadOnlyCollectionBuilder<T>.Reverse.

readOnlyCollection.AsEnumerable().Reverse(); // Enumerable.Reverse.

IQueryable<int> queryable = new EnumerableQuery<int>(Enumerable.Empty<int>());

queryable.Reverse(); // Queryable.Reverse.

queryable.AsEnumerable().Reverse(); // Enumerable.Reverse.

ImmutableList<int> immutableList = ImmutableList.Create(0);

immutableList.Reverse(); // ImmutableSortedSet<T>.Reverse.

immutableList.AsEnumerable().Reverse(); // Enumerable.Reverse.

ImmutableSortedSet<int> immutableSortedSet = ImmutableSortedSet.Create(0);

immutableSortedSet.Reverse(); // ImmutableSortedSet<T>.Reverse.

immutableSortedSet.AsEnumerable().Reverse(); // Enumerable.Reverse.

}

As fore mentioned, local parallel LINQ queries are represented by ParallelQuery<T> and remote LINQ queries are represented by IQueryable<T>. They both implement IEnumerable<T>, so they both have AsEnumerable available. Since AsEnumerable returns IEnumerable<T>, it opt-out local parallel query and remote query back to local sequential query. These scenarios are discussed in Parallel LINQ chapter and LINQ to Entities chapter.

Collection queries

The collection query methods all convert source sequence to a collection by pulling all the values from the source sequence and store them in array, list, dictionary, or lookup.

Conversion

ToArray and ToList are straightforward:

public static TSource[] ToArray<TSource>(this IEnumerable<TSource> source);

public static List<TSource> ToList<TSource>(this IEnumerable<TSource> source);

They pull all values from the source sequence, and simply store them into a new array/list:

internal static void ToArrayToList()

int[] array = Enumerable

.Range(0, 5) // Define query, return IEnumerable<T>.

.ToArray(); // Execute query.

List<int> list = Enumerable

.Range(0, 5) // Define query, return IEnumerable<T>.

.ToList(); // Execute query.

}

Apparently, when collection query methods are called for an IEnumerable<T> sequence representing LINQ query, that LINQ query is executed immediately. Similarly, ToDictionary/ToLookup also pulls all values from source sequence, and store those values into a new dictionary/lookup:

public static Dictionary<TKey, TSource> ToDictionary<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static ILookup<TKey, TSource> ToLookup<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static Dictionary<TKey, TElement> ToDictionary<TSource, TKey, TElement>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector, Func<TSource, TElement> elementSelector);

public static ILookup<TKey, TElement> ToLookup<TSource, TKey, TElement>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector, Func<TSource, TElement> elementSelector);

Here are the definition of dictionary and lookup:

namespace System.Collections.Generic

public class Dictionary<TKey, TValue>: IEnumerable<KeyValuePair<TKey, TValue>>, IEnumerable,

IDictionary<TKey, TValue>, IDictionary, ICollection<KeyValuePair<TKey, TValue>>, ICollection,

IReadOnlyDictionary<TKey, TValue>, IReadOnlyCollection<KeyValuePair<TKey, TValue>>,

ISerializable, IDeserializationCallback { }

namespace System.Linq

public interface ILookup<TKey, TElement>: IEnumerable<IGrouping<TKey, TElement>>, IEnumerable

IEnumerable<TElement> this[TKey key] { get; }

int Count { get; }

bool Contains(TKey key);

}

The difference between dictionary and lookup is, a dictionary is a flatten collection of key-value pairs, where each key is paired with one single value, and a lookup is a hierarchical collection of key-sequence pairs, where each key is a sequence of paired with one or more values.

internal static void ToDictionaryToLookup()

Dictionary<int, string> dictionary = Enumerable

.Range(0, 5) // Define query.

.ToDictionary(

keySelector: int32 => int32,

elementSelector: int32 => Math.Sqrt(int32).ToString("F", CultureInfo.InvariantCulture)); // Execute query.

foreach (KeyValuePair<int, string> squareRoot in dictionary)

$"√{squareRoot.Key}:{squareRoot.Value}".WriteLine();

// √0: 0.00

// √1: 1.00

// √2: 1.41

// √3: 1.73

// √4: 2.00

ILookup<int, int> lookup = Enumerable

.Range(-2, 5) // Define query.

.ToLookup(int32 => int32 * int32); // Execute query.

foreach (IGrouping<int, int> squareRoots in lookup)

$"√{squareRoots.Key}: ".Write();

foreach (int squareRoot in squareRoots)

$"{squareRoot}, ".Write();

Environment.NewLine.Write();

// √4: -2, 2,

// √1: -1, 1,

// √0: 0,

}

Each value from source sequence is mapped to a key by calling the key selector function. If element selector is provided, each value from source sequence is mapped to a value in the result dictionary/lookup. In above example, if ToDictionary is called in the second query, an ArgumentException is thrown because dictionary cannot have multiple key and single value pairs with the same key:

internal static void ToDictionaryWithException()

Dictionary<int, int> lookup = Enumerable

.Range(-2, 5) // Define query.

.ToDictionary(int32 => int32 * int32); // Execute query.

// ArgumentException: An item with the same key has already been added.

}

Another difference between dictionary and lookup is, at runtime, if querying a dictionary with a non-existing key, dictionary throws KeyNotFoundException, but if querying a lookup with a non-existing key, lookup returns an empty sequence peacefully.

internal static void LookupDictionary()

ILookup<int, int> lookup = Enumerable

.Range(0, 5) // Define query.

.ToLookup(int32 => int32); // Execute query.

int count = 0;

IEnumerable<int>group = lookup[10];

foreach (int value in group)

count++;

count.WriteLine(); // 0

Dictionary<int, int> dictionary = Enumerable

.Range(0, 5) // Define query.

.ToDictionary(int32 => int32); // Execute query.

int result = dictionary[10];

// KeyNotFoundException: The given key was not present in the dictionary.

}

The last difference is dictionary cannot have null key, while lookup can:

internal static void LookupDictionaryNullKey()

ILookup<string, string> lookup = new string[] { "a", "b", null }.ToLookup(@string => @string);

int count = 0;

IEnumerable<string>group = lookup[null];

foreach (string value in group)

count++;

count.WriteLine(); // 1

Dictionary<string, string>dictionary = new string[] { "a", "b", null }

.ToDictionary(@string => @string);

// ArgumentNullException: Value cannot be null. Parameter name: key.

}

ToDictionary/ToLookup has other overloads to accept a key comparer:

public static Dictionary<TKey, TSource> ToDictionary<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector, IEqualityComparer<TKey> comparer);

public static ILookup<TKey, TSource> ToLookup<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey>keySelector, IEqualityComparer<TKey> comparer);

public static Dictionary<TKey, TElement> ToDictionary<TSource, TKey, TElement>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

IEqualityComparer<TKey>comparer);

public static ILookup<TKey, TElement> ToLookup<TSource, TKey, TElement>(

this IEnumerable<TSource> source,

Func<TSource, TKey> keySelector,

Func<TSource, TElement> elementSelector,

IEqualityComparer<TKey>comparer);

For example:

internal static void ToLookupWithComparer()

ILookup<string, string> lookup = new string[] { "aa", "AA", "Aa", "aA", "bb" }

.ToLookup(@string => @string, StringComparer.OrdinalIgnoreCase);

foreach (IGrouping<string, string> group in lookup)

$"{group.Key}: ".Write();

foreach (string @string in group)

$"{@string}, ".Write();

Environment.NewLine.Write();

// aa: aa, AA, Aa, aA,

// bb: bb,

}

Value queries

These query methods pull single, partial, or all values of the source sequence to output a specified value or calculate a result with the pulled values.

Element

Element query methods returns a single value from the source sequence. When they are called, they immediately execute the query, trying to pull values until the expected value is pulled. First/Last immediately pulls the first/last value of the source sequence.

public static TSource First<TSource>(this IEnumerable<TSource> source);

public static TSource Last<TSource>(this IEnumerable<TSource> source);

And InvalidOperationException is thrown if the source sequence is empty.

internal static IEnumerable<int> Int32Source() => new int[] { -1, 1, 2, 3, -4 };

internal static IEnumerable<int>SingleInt32Source() => Enumerable.Repeat(5, 1);

internal static IEnumerable<int>EmptyInt32Source() => Enumerable.Empty<int>();

internal static void FirstLast()

int firstOfSource = Int32Source().First().WriteLine(); // -1

int lastOfSource = Int32Source().Last().WriteLine(); // -4

int firstOfSingleSource = SingleInt32Source().First().WriteLine(); // 5

int lastOfSingleSource = SingleInt32Source().Last().WriteLine(); // 5

int firstOfEmptySource = EmptyInt32Source().First(); // InvalidOperationException.

int lastOfEmptySource = EmptyInt32Source().Last(); // InvalidOperationException.

}

The other First/Last overload accept a predicate function. They immediately call the predicate function immediately with the values, and return the first/last value where predicate function returns true:

public static TSource First<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate);

public static TSource Last<TSource>(this IEnumerable<TSource> source, Func<TSource, bool>predicate);

Logically, source.First(predicate) is equivalent to source.Where(predicate).First(), and source.Last(predicate) is equivalent to source.Where(predicate).Last():

internal static void FirstLastWithPredicate()

int firstPositiveOfSource = Int32Source().First(int32 => int32 > 0).WriteLine(); // 1

int lastNegativeOfSource = Int32Source().Last(int32 => int32 < 0).WriteLine(); // -4

int firstPositiveOfSingleSource = SingleInt32Source().First(int32 => int32 > 0).WriteLine(); // 1

int lastNegativeOfSingleSource = SingleInt32Source().Last(int32 => int32 < 0); // InvalidOperationException.

int firstPositiveOfEmptySource = EmptyInt32Source().First(int32 => int32 > 0); // InvalidOperationException.

int lastNegativeOfEmptySource = EmptyInt32Source().Last(int32 => int32 < 0); // InvalidOperationException.

}

There are also FirstOrDefault/LastOrDefault methods:

public static TSource FirstOrDefault<TSource>(this IEnumerable<TSource> source);

public static TSource FirstOrDefault<TSource>(this IEnumerable<TSource> source, Func<TSource, bool>predicate);

public static TSource LastOrDefault<TSource>(this IEnumerable<TSource> source);

public static TSource LastOrDefault<TSource>(this IEnumerable<TSource> source, Func<TSource, bool>predicate);

When there is no first/last value available, these methods return a default value instead of throwing exception:

internal static void FirstOrDefaultLastOrDefault()

int firstOrDefaultOfEmptySource = EmptyInt32Source().FirstOrDefault().WriteLine(); // 0

int lastOrDefaultOfEmptySource = EmptyInt32Source().LastOrDefault().WriteLine(); // 0

int lastNegativeOrDefaultOfSingleSource = SingleInt32Source().LastOrDefault(int32 => int32 < 0).WriteLine(); // 0

int firstPositiveOrDefaultOfEmptySource = EmptyInt32Source().FirstOrDefault(int32 => int32 > 0).WriteLine(); // 0

int lastNegativeOrDefaultOfEmptySource = EmptyInt32Source().LastOrDefault(int32 => int32 < 0).WriteLine(); // 0

Character lokiOrDefault = Characters()

.FirstOrDefault(character => "Loki".Equals(character.Name, StringComparison.Ordinal));

(lokiOrDefault == null).WriteLine(); // True

}

ElementAt returns the value at the specified index:

public static TSource ElementAt<TSource>(this IEnumerable<TSource> source, int index);

When the specified index is out of range, ArgumentOutOfRangeException is thrown.

internal static void ElementAt()

int elementAt2OfSource = Int32Source().ElementAt(2).WriteLine(); // 2

int elementAt9OfSource = Int32Source().ElementAt(9); // ArgumentOutOfRangeException.

int elementAtNegativeIndex = Int32Source().ElementAt(-5); // ArgumentOutOfRangeException.

int elementAt0OfSingleSource = SingleInt32Source().ElementAt(0).WriteLine(); // 5

int elementAt1OfSingleSource = SingleInt32Source().ElementAt(1); // ArgumentOutOfRangeException.

int elementAt0OfEmptySource = EmptyInt32Source().ElementAt(0); // ArgumentOutOfRangeException.

}

Similarly, there is ElementAtOrDefault:

public static TSource ElementAtOrDefault<TSource>(this IEnumerable<TSource> source, int index);

When there is no value available at the specified index, a default value is returned:

internal static void ElementAtOrDefault()

int elementAt9OrDefaultOfSource = Int32Source().ElementAtOrDefault(9).WriteLine(); // 0

int elementAtNegativeIndexOrDefault = Int32Source().ElementAtOrDefault(-5).WriteLine(); // 0

int elementAt1OrDefaultOfSingleSource = SingleInt32Source().ElementAtOrDefault(1).WriteLine(); // 0

int elementAt0OrDefaultOfEmptySource = EmptyInt32Source().ElementAtOrDefault(0).WriteLine(); // 0

Character characterAt5OrDefault = Characters().ElementAtOrDefault(5);

(characterAt5OrDefault == null).WriteLine(); // True

}

Single is more strict. It pulls the single value from a singleton sequence.

public static TSource Single<TSource>(this IEnumerable<TSource> source);

public static TSource Single<TSource>(this IEnumerable<TSource> source, Func<TSource, bool>predicate);

If source sequence has no value or has more than one values, InvalidOperationException is thrown:

internal static void Single()

int singleOfSource = Int32Source().Single(); // InvalidOperationException.

int singleGreaterThan2OfSource = Int32Source().Single(int32 => int32 > 2).WriteLine(); // 3

int singleNegativeOfSource = Int32Source().Single(int32 => int32 < 0); // InvalidOperationException.

int singleOfSingleSource = SingleInt32Source().Single().WriteLine(); // 5

int singleNegativeOfSingleSource = SingleInt32Source().Single(int32 => int32 < 0); // InvalidOperationException.

int singleOfEmptySource = EmptyInt32Source().Single(); // InvalidOperationException.

int singlePositiveOfEmptySource = EmptyInt32Source().Single(int32 => int32 == 0); // InvalidOperationException.

Character singleCharacter = Characters().Single(); // InvalidOperationException.

Character fromAsgard = Characters()

.Single(character => "Asgard".Equals(character.PlaceOfBirth, StringComparison.Ordinal))

.WriteLine(); // Thor

Character loki = Characters().Single(

character => "Loki".Equals(character.Name, StringComparison.Ordinal)); // InvalidOperationException.

}

SingleOrDefault is just slightly less strict than Single:

public static TSource SingleOrDefault<TSource>(this IEnumerable<TSource> source);

public static TSource SingleOrDefault<TSource>(this IEnumerable<TSource>source, Func<TSource, bool> predicate);

When source sequence has no value, it returns a default value. When source sequence has more than one values, it still throws InvalidOperationException:

internal static void SingleOrDefault()

int singleOrDefaultOfSource = Int32Source().SingleOrDefault(); // InvalidOperationException.

int singleNegativeOrDefaultOfSource = Int32Source().SingleOrDefault(int32 => int32 < 0); // InvalidOperationException.

int singleNegativeOrDefaultOfSingleSource = SingleInt32Source().SingleOrDefault(int32 => int32 < 0).WriteLine(); // 0

int singleOrDefaultOfEmptySource = EmptyInt32Source().SingleOrDefault().WriteLine(); // 0

int singlePositiveOrDefaultOfEmptySource = EmptyInt32Source().SingleOrDefault(int32 => int32 == 0); // 0

Character singleCharacterOrDefault = Characters().SingleOrDefault(); // InvalidOperationException.

Character lokiOrDefault = Characters()

.SingleOrDefault(character => "Loki".Equals(character.Name, StringComparison.Ordinal));

(lokiOrDefault == null).WriteLine(); // True

}

Aggregation

Aggregate query methods pull all values from source sequence, and repeatedly call a function to accumulate those value. The easiest overload accepts an accumulator function:

public static TSource Aggregate<TSource>(this IEnumerable<TSource> source, Func<TSource, TSource, TSource> func);

Aggregate requires the source sequence to be not empty. When the source sequence is empty, it throws InvalidOperationException. When there is only 1 single value in the source sequence, it returns that value. When there are more than 1 values, it calls the accumulator function to accumulate the first and second values to a result, and then call the accumulator function again to accumulate the previous result and the third value to another result, and so on, until all values are accumulated, eventually it returns the result of the last accumulator function call.

internal static void Aggregate()

int productOfSource = Int32Source()

.Aggregate((currentProduct, int32) => currentProduct * int32)

.WriteLine(); // ((((-1 * 1) * 2) * 3) * -4) = 24.

int productOfSingleSource = SingleInt32Source()

.Aggregate((currentProduct, int32) => currentProduct * int32).WriteLine(); // 5

int productOfEmptySource = EmptyInt32Source()

.Aggregate((currentProduct, int32) => currentProduct * int32); // InvalidOperationException.

}

There is another overload accepts a seed:

public static TAccumulate Aggregate<TSource, TAccumulate>(this IEnumerable<TSource> source, TAccumulate seed, Func<TAccumulate, TSource, TAccumulate> func);

With the seed provided, Aggregate does not require the source sequence to be not empty. When the source sequence is empty, it returns the seed. When the source sequence is not empty, it calls the accumulator function to accumulate the seed value and the first values to a result, and then call the accumulator function again to accumulate the previous result and the second to another result, and so on, until all values are accumulated, eventually it also returns the result of the last accumulator function call.

internal static void AggregateWithSeed()

int sumOfSquaresOfSource = Int32Source()

.Aggregate(

seed: 0,

func: (currentSumOfSquares, int32) => currentSumOfSquares + int32 * int32)

.WriteLine(); // 31

int sumOfSquaresOfSingleSource = SingleInt32Source()

.Aggregate(

seed: 0,

func: (currentSumOfSquares, int32) => currentSumOfSquares + int32 * int32)

.WriteLine(); // 25

int sumOfSquaresOfEmptySource = EmptyInt32Source()

.Aggregate(

seed: 0,

func: (currentSumOfSquares, int32) => currentSumOfSquares + int32 * int32)

.WriteLine(); // 0

}

The last overload accepts an additional result selector function, which is called with the last result of accumulate function:

internal static TResult Aggregate<TSource, TAccumulate, TResult>(

this IEnumerable<TSource> source,

TAccumulate seed,

Func<TAccumulate, TSource, TAccumulate>func, Func<TAccumulate, TResult> resultSelector);

So, source.Aggregate(seed, accumulation, resultSelector) is equivalent to resultSelector(source.Aggregate(seed, accumulation)):

internal static void AggregateWithSeedAndResultSelector()

string sumOfSquaresMessage = Int32Source()

.Aggregate(

seed: 0,

func: (currentSumOfSquares, int32) => currentSumOfSquares + int32 * int32,

resultSelector: result => $"Sum of squares: {result}")

.WriteLine(); // Sum of squares: 31

}

Count returns the number of values in source sequence:

public static int Count<TSource>(this IEnumerable<TSource> source);

It is one of the most intuitive query methods:

internal static void Count()

int countOfSource = Int32Source().Count().WriteLine(); // 5

int countOfSingleSource = SingleInt32Source().Count().WriteLine(); // 1

int countOfEmptySource = EmptyInt32Source().Count().WriteLine(); // 0

int countOfCharacters = Characters().Count().WriteLine(); // 5

int countOfTypesInCoreLibrary = CoreLibrary.ExportedTypes.Count().WriteLine(); // 1523

}

The other overload accepts a predicate:

public static int Count<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate);

Similar to First/Last, source.Count(predicate) is equivalent to ource.Where(predicate).Count():

internal static void CountWithPredicate()

int positiveCountOfSource = Int32Source().Count(int32 => int32 > 0).WriteLine(); // 3

int positiveCountOfSingleSource = SingleInt32Source().Count(int32 => int32 > 0).WriteLine(); // 1

int positiveCountOfEmptySource = EmptyInt32Source().Count(int32 => int32 > 0).WriteLine(); // 0

int countOfConcat = Enumerable

.Repeat(0, int.MaxValue)

.Concat(Enumerable.Repeat(0, int.MaxValue))

.Count(); // OverflowException.

int countOfCharactersFromUS = Characters()

.Count(character => "US".Equals(character.PlaceOfBirth, StringComparison.Ordinal))

.WriteLine(); // 3

}

LongCount is similar to Count. It can be used for large sequence, and returns a long (System.Int64) value instead of int (System.Int32):

public static long LongCount<TSource>(this IEnumerable<TSource> source);

public static long LongCount<TSource>(this IEnumerable<TSource>source, Func<TSource, bool> predicate);

For example:

internal static void LongCount()

long longCountOfSource = Int32Source().LongCount().WriteLine(); // 5L

long countOfConcat = Enumerable

.Repeat(0, int.MaxValue)

.Concat(Enumerable.Repeat(0, int.MaxValue))

.LongCount()

.WriteLine(); // int.MaxValue + int.MaxValue = 4294967294L

}

Max/Min also pulls all values from the source sequence of int values, and returns the minimum/maximum value:

public static int Max(this IEnumerable<int> source);

public static int Min(this IEnumerable<int> source);

Max/Min throw InvalidOperationException if the source sequence is empty:

internal static void MinMax()

int minOfSource = Int32Source().Min().WriteLine(); // -4

int maxOfSource = Int32Source().Max().WriteLine(); // 3

int minOfSingleSource = SingleInt32Source().Min().WriteLine(); // 5

int maxOfSingleSource = SingleInt32Source().Max().WriteLine(); // 5

int minOfEmptySource = EmptyInt32Source().Min(); // InvalidOperationException.

int maxOfEmptySource = EmptyInt32Source().Max(); // InvalidOperationException.

}

The other overload accepts a sequence of arbitrary type, and a selector function which maps each value to an int value for comparison:

public static int Max<TSource>(this IEnumerable<TSource> source, Func<TSource, int> selector);

public static int Min<TSource>(this IEnumerable<TSource>source, Func<TSource, int> selector);

The following example queries the maximum type (type with the largest number of public members declared) in the .NET core library:

internal static void MaxWithSelector()

int mostDeclaredMembers = CoreLibrary.ExportedTypes

.Max(type => type.GetDeclaredMembers().Length).WriteLine(); // 311

}

Here each public type is mapped the count of its public members’ count number. The maximum type in .NET core library has 311 public members. Here Max returns the maximum count of members, but does not tell which type is that count from. To query the maximum type along with the member count, Aggregate can be used to pull all types and accumulate by the maximum member count:

internal static void AggregateWithAnonymousTypeSeed()

(List<Type>Types, int MaxMemberCount) maxTypes = CoreLibrary.ExportedTypes.Aggregate(

seed: (Types: new List<Type>(), MaxMemberCount: 0),

func: (currentMax, type) =>

List<Type>currentMaxTypes = currentMax.Types;

int currentMaxMemberCount = currentMax.MaxMemberCount;

int memberCount = type.GetDeclaredMembers().Length;

if (memberCount > currentMaxMemberCount)

currentMaxTypes.Clear();

currentMaxTypes.Add(type);

currentMaxMemberCount = memberCount;

else if (memberCount == currentMaxMemberCount)

// If multiple types have the same maximum member count, take all those types.

currentMaxTypes.Add(type);

return (Types: currentMaxTypes, MaxMemberCount: currentMaxMemberCount);

}); // Define query.

maxTypes.Types.WriteLines(maxType => $"{maxType.FullName}:{maxTypes.MaxMemberCount}");

// Execute query. System.Convert:311

}

In the core library, System.Convert is the winner, with 311 public members declared.

Besides int, Max/Min has overloads for int?, long, long?, double, double?, float, float?, decimal, decimal?. There are also overloads for arbitrary comparable type:

public static TSource Max<TSource>(this IEnumerable<TSource> source);

public static TSource Min<TSource>(this IEnumerable<TSource> source);

They use Comparer<TSource>.Default to compare values in source sequence to determine the minimum/maximum value. Comparer<TSource>.Default requires TSource to implement at least one of IComparable and IComparable<TSource>; otherwise ArgumentException is thrown at runtime. Still take Character type as example:

internal partial class Character : IComparable<Character>

public int CompareTo(Character other) =>

string.Compare(this.Name, other.Name, StringComparison.Ordinal);

}

Now Max/Min can be used with character sequence:

internal static void MaxMinGeneric()

Character maxCharacter = Characters().Max().WriteLine(); // Vision

Character minCharacter = Characters().Min().WriteLine(); // JAVIS

}

Max/Min also have overload for arbitrary type, with a selector function to maps each value to a comparable result:

public static TResult Max<TSource, TResult>(this IEnumerable<TSource> source, Func<TSource, TResult> selector);

public static TResult Min<TSource, TResult>(this IEnumerable<TSource>source, Func<TSource, TResult> selector);

For example:

internal static void MaxMinGenericWithSelector()

string maxName = Characters().Max(character => character.Name).WriteLine(); // Vision

string minName = Characters().Min(character => character.Name).WriteLine(); // JAVIS

}

Apparently, source.Max(selector) is equivalent to source.Select(selector),Max, and source.Min(selector) is equivalent to source.Select(selector).Min().

Sum/Average pulls all int values from the source sequence, and calculate the sum/average of all the values. The signatures are similar to Max/Min:

public static int Sum(this IEnumerable<int> source);

public static double Average(this IEnumerable<int> source);

Here Average returns double instead of int. Also, when called with empty source sequence, Sum returns 0, while Average throws InvalidOperationException:

internal static void SumAverage()

int sumOfSource = Int32Source().Sum().WriteLine(); // 1

double averageOfSource = Int32Source().Average().WriteLine(); // 0.2

int sumOfSingleSource = SingleInt32Source().Sum().WriteLine(); // 5

double averageOfSingleSource = SingleInt32Source().Average().WriteLine(); // 5.0

int sumOfEmptySource = EmptyInt32Source().Sum().WriteLine(); // 0

double averageOfEmptySource = EmptyInt32Source().Average().WriteLine(); // InvalidOperationException.

}

Sum/Average has overload for arbitrary type, with a selector function to map each value to int value for calculation:

public static int Sum<TSource>(this IEnumerable<TSource> source, Func<TSource, int> selector);

public static double Average<TSource>(this IEnumerable<TSource>source, Func<TSource, int> selector);

The following example calculate the average count of public members declared on types in the core library, and the average count of all public members.

internal static void AverageWithSelector()

double averageMemberCount = CoreLibrary.ExportedTypes

.Average(type => type.GetMembers().Length)

.WriteLine(); // 22.0766378244747

double averageDeclaredMemberCount = CoreLibrary.ExportedTypes

.Average(type => type.GetDeclaredMembers().Length)

.WriteLine(); // 11.7527812113721

}

Similarly, Sum/Average also has overloads for int?, long, long?, double, double?, float, float?, decimal, decimal?.

Quantifier

Any determines whether the source sequence is not empty, by immediately trying to pull the first value from source sequence:

public static bool Any<TSource>(this IEnumerable<TSource> source);

For example.

internal static void Any()

bool anyInSource = Int32Source().Any().WriteLine(); // True

bool anyInSingleSource = SingleInt32Source().Any().WriteLine(); // True

bool anyInEmptySource = EmptyInt32Source().Any().WriteLine(); // False

}

The other overload accepts a predicate function.

public static bool Any<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate);

Logically, source.Any(predicate) is equivalent to source.Where(predicate).Any().

internal static void AnyWithPredicate()

bool anyNegative = Int32Source().Any(int32 => int32 < 0).WriteLine(); // True

bool anyPositive = SingleInt32Source().Any(int32 => int32 > 0).WriteLine(); // True

bool any0 = EmptyInt32Source().Any(_ => true).WriteLine(); // False

}

All accepts a predicate. It also tries to pull values from the source sequence, and calls predicate function with each value. It returns true if predicate returns true for all values; otherwise, it returns false:

public static bool All<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate);

All always returns true for empty source.

internal static void All()

bool allNegative = Int32Source().All(int32 => int32 < 0).WriteLine(); // False

bool allPositive = SingleInt32Source().All(int32 => int32 > 0).WriteLine(); // True

bool allGreaterThanMax = EmptyInt32Source().All(int32 => int32 > int.MaxValue).WriteLine(); // True

}

Contains determines whether source sequence contains the specified value:

public static bool Contains<TSource>(this IEnumerable<TSource> source, TSource value);

For example:

internal static void Contains()

bool contains5InSource = Int32Source().Contains(5).WriteLine(); // False

bool contains5InSingleSource = SingleInt32Source().Contains(5).WriteLine(); // True

bool contains5InEmptySource = EmptyInt32Source().Contains(5).WriteLine(); // False

}

The other overload of Contains accepts a comparer:

public static bool Contains<TSource>(

this IEnumerable<TSource> source, TSource value, IEqualityComparer<TSource> comparer);

For example:

internal static void ContainsWithComparer()

bool containsTwo = Words().Contains("two", StringComparer.Ordinal).WriteLine(); // False

bool containsTwoIgnoreCase = Words().Contains("two", StringComparer.OrdinalIgnoreCase).WriteLine(); // True

}

Similar to other query methods, the first overload without comparer uses EqualityComparer<TSource>.Default.

Equality

.NET has many ways to determine equality for objects:

· Reference equality/identity: object.ReferenceEquals, == operator without override

· Value equality/equivalence: static object.Equals, instance object.Equals, object.GetHashCode, overridden == operator, IEquatable<T>.Equals, IEqualityComparer.Equals, IEqualityComparer<T>.Equals, IComparable.Compare, IComparable<T>.Compare, IComparer.Compare, IComparer<T>.Compare

· Sequential equality: Enumerable.SequentialEqual

SequentialEqual query method is provided to compares the sequential equality of 2 IEnumerable<T> sequences:

public static bool SequenceEqual<TSource>(this IEnumerable<TSource> first, IEnumerable<TSource> second);

2 sequences are sequentially equal if their length is equal, and for each index, 2 values from both sequences are equal (determined by EqualityComparer<TSource>.Default).

internal static void SequentialEqual()

IEnumerable<object>first = new object[] { null, 1, "2", CoreLibrary };

IEnumerable<object>second = new List<object>() { null, 1, $"{1 + 1}", CoreLibrary };

bool valueEqual = first.Equals(second).WriteLine(); // False

bool referenceEqual = object.ReferenceEquals(first, second).WriteLine(); // False

bool sequentialEqual = first.SequenceEqual(second.Concat(Enumerable.Empty<object>())).WriteLine(); // True

}

Empty sequences with the same TSource type are sequentially equal:

internal static void SequentialEqualOfEmpty()

IEnumerable<Derived>emptyFirst = new ConcurrentQueue<Derived>();

IEnumerable<Base>emptySecond = ImmutableHashSet.Create<Base>();

bool sequentialEqual = emptyFirst.SequenceEqual(emptySecond).WriteLine(); // True

}

The other overload accepts a comparer:

public static bool SequenceEqual<TSource>(

this IEnumerable<TSource> first, IEnumerable<TSource> second, IEqualityComparer<TSource> comparer);

For example:

internal static void SequentialEqualWithComparer()

IEnumerable<string>first = new string[] { null, string.Empty, "ss", };

IEnumerable<string>second = new string[] { null, string.Empty, "ß", };

CultureInfo.CurrentCulture = new CultureInfo("en-US");

bool sequentialEqual1 = first.SequenceEqual(second, StringComparer.CurrentCulture).WriteLine(); // True

bool sequentialEqual2 = first.SequenceEqual(second, StringComparer.Ordinal).WriteLine(); // False

}

Again, the first overload without comparer uses EqualityComparer<TSource>.Default.

Summary

This chapter discusses IEnumerable<T> interface, which represents LINQ to Objects source or query, then discusses each standard query in query method syntax and query expression syntax if applicable. With this detailed tutorial, reader should be able to master how to use LINQ to Objects to query data objects in memory.

LINQ to Objects in Depth (1) Local Sequential Query

Mon, 01 Jul 2019 00:00:00 GMT

[LINQ via C# series]

[LINQ to Objects in Depth series]

LINQ to Objects queries .NET objects in local memory of current application or service. Its data source and the queries are represented by IEnumerable<T> interface, and it is executed sequentially. This chapter introduces IEnumerable<T> interface, and discusses how to use LINQ to Object to query objects in query method syntax and query expression syntax.

Local sequential LINQ query

LINQ to Objects is local, sequential query. Local means it queries data is instances of .NET types available in the memory of current .NET application or service, not remote data like rows in a data table managed by a separate database. In .NET, IEnumerable<T> interface represents a sequence of instances of T type, so when LINQ is introduced to .NET, IEnumerable<T> is naturally reused to represent LINQ to Objects data source and query. Sequential means when LINQ to Objects query is executed, the instances from data source are processed one after another in a single threading manner instead of parallelism. In another word, LINQ to Objects query results can be pulled from IEnumerable<T> one by one.

Iterator pattern and foreach statement

If a sequence type is defined following the standard iterator pattern of object-oriented programming, the objects in the sequence can be pulled by C# foreach statement. Iterator pattern consists of a sequence (also called container of items, or aggregate of elements) and an iterator:

internal abstract class Sequence

public abstract Iterator GetEnumerator(); // Must be public.

internal abstract class Iterator

public abstract bool MoveNext(); // Must be public.

public abstract object Current { get; } // Must be public.

}

And their generic version is:

internal abstract class GenericSequence<T>

public abstract GenericIterator<T>GetEnumerator(); // Must be public.

internal abstract class GenericIterator<T>

public abstract bool MoveNext(); // Must be public.

public abstract T Current { get; } // Must be public.

}

The above types and members demonstrate the minimum requirements of iterator pattern for foreach statement. The sequence must have a GetEnumerator factory method to output an iterator (It can be virtually read as GetIterator). And iterator must have a MoveNext method to output a bool value to indicate whether there is a value that can be pulled. If the output is true, iterator’s Current property getter can be called to pull that value. Now the values in above non-generic and generic sequences can be access with C# foreach statement:

internal static void ForEach<T>(Sequence sequence, Action<T> process)

foreach (T value in sequence)

process(value);

internal static void ForEach<T>(GenericSequence<T> sequence, Action<T> process)

foreach (T value in sequence)

process(value);

}

The foreach statement is declarative syntactic sugar. It is compiled to the following imperative control flow to get iterator from sequence, and poll iterator until there is no value available:

internal static void CompiledForEach<T>(Sequence sequence, Action<T> process)

Iterator iterator = sequence.GetEnumerator();

try

while (iterator.MoveNext())

T value = (T)iterator.Current;

process(value);

finally

(iterator as IDisposable)?.Dispose();

internal static void CompiledForEach<T>(GenericSequence<T> sequence, Action<T> process)

GenericIterator<T>iterator = sequence.GetEnumerator();

try

while (iterator.MoveNext())

T value = iterator.Current;

process(value);

finally

(iterator as IDisposable)?.Dispose();

}

Apparently, the generic version of definition is preferred, because the non-generic iterator’s Current property outputs object type, which has to be explicitly cast to the expected type specified in the foreach statement and could be a chance of failure. To demonstrate the iterator pattern implementation, a sequence of values can be defined as a singly linked list, where each value is stored in a linked list node:

internal class LinkedListNode<T>

internal LinkedListNode(T value, LinkedListNode<T> next = null) =>

(this.Value, this.Next) = (value, next);

public T Value { get; }

public LinkedListNode<T> Next { get; }

}

Then iterator can be implemented to traverse along the linked list nodes. When there is a next node, MoveNext outputs true, and Current can output the next value. Notice the iterator is stateful:

internal class LinkedListIterator<T> : GenericIterator<T>

private LinkedListNode<T> node; // State.

internal LinkedListIterator(LinkedListNode<T>head) =>

this.node = new LinkedListNode<T>(default, head);

public override bool MoveNext()

if (this.node.Next != null)

this.node = this.node.Next; // State change.

return true;

return false;

public override T Current => this.node.Value;

}

Finally, the sequence can be simply implemented as an iterator factory:

internal class LinkedListSequence<T> : GenericSequence<T>

private readonly LinkedListNode<T> head;

internal LinkedListSequence(LinkedListNode<T> head) => this.head = head;

public override GenericIterator<T> GetEnumerator() => new LinkedListIterator<T>(this.head);

}

Now the values in the linked list sequence can be sequentially pulled with the foreach syntactic sugar, which is declarative and there is no need to specify the control flow or handle the state:

internal static void ForEach()

LinkedListNode<int> node3 = new LinkedListNode<int>(3, null);

LinkedListNode<int> node2 = new LinkedListNode<int>(2, node3);

LinkedListNode<int> node1 = new LinkedListNode<int>(1, node2);

LinkedListSequence<int> sequence = new LinkedListSequence<int>(node1);

foreach (int value in sequence)

value.WriteLine(); // 1 2 3

}

A general implementation of iterator pattern is discussed later in the LINQ to Objects query implementation chapter.

IEnumerable<T> and IEnumerator<T> interfaces

Initially, .NET Framework 1.0 provides IEnumerable and IEnumerator interfaces as the contract of iterator pattern:

namespace System.Collections

public interface IEnumerable // Sequence.

IEnumerator GetEnumerator();

public interface IEnumerator // Iterator.

object Current { get; }

bool MoveNext();

void Reset(); // For COM interoperability.

}

They can be virtually read as iteratable sequence and iterator. .NET Framework’s sequence and collection types implement IEnumerable so that they can be used with foreach statement, like array, ArrayList, Queue, Stack, SortedList, etc. Then .NET Framework 2.0 was released with generics support. So IEnumerable<T> and IEnumerator<T> interfaces are provided as the generic version of iterator pattern contract.

namespace System.Collections.Generic

public interface IEnumerable<T> : IEnumerable // Sequence.

IEnumerator<T> GetEnumerator();

public interface IEnumerator<T> : IDisposable, IEnumerator // Iterator.

T Current { get; }

}

Since .NET Framework 2.0, sequence and collection types are usually provided as generic types, with IEnumerable<T> implemented, like array, List<T>, Queue<T>, Stack<T>, SortedList<T>, etc.

Later, .NET Framework 4.0 introduces covariance and contravariance for generic interface. As discussed in the covariance and contravariance chapter, T is covariant for both IEnumerable<T> and IEnumerable<T>. So, their definitions are updated with the out modifier for type parameter:

namespace System.Collections.Generic

public interface IEnumerable<out T> : IEnumerable // Sequence.

IEnumerator<T> GetEnumerator();

public interface IEnumerator<out T> : IDisposable, IEnumerator // Iterator.

T Current { get; }

}

This is how they are defined in .NET Standard.

foreach statement vs. for statement

Array is a special type. A concrete array T[] inherits System.Array type, which does not implement IEnumerable<T> but IEnumerable:

namespace System

public abstract class Array : ICollection, IEnumerable, IList, IStructuralComparable, IStructuralEquatable

}

Instead, T[] directly implements IEnumerable<T>, ICollection<T>, and IList<T>, as long as T[] is single dimensional and zero–lower bound. So, array T[] can be used with foreach loop:

internal static void ForEach<T>(T[] array, Action<T> process)

foreach (T value in array)

process(value);

}

foreach statement with array is compiled into a for loop, accessing each value with index, which has better performance than the above control flow of getting iterator and polling its MoveNext method and Current property:

internal static void CompiledForEach<T>(T[] array, Action<T> process)

for (int index = 0; index < array.Length; index++)

T value = array[index];

process(value);

}

And so is string. Since string is a sequence of characters, it implements IEnumerable<char>. Foreach statement with string is also compiled to index access for better performance:

internal static void ForEach(string @string, Action<char> process)

foreach (char value in @string)

process(value);

internal static void CompiledForEach(string @string, Action<char> action)

for (int index = 0; index < @string.Length; index++)

char value = @string[index];

action(value);

}

LINQ to Objects queryable types

As discussed, most LINQ to Objects queries are extension methods for IEnumerable<T>. Since most of.NET sequence and collection types implements IEnumerable<T>, LINQ to Object queries are available for these types, and foreach statement is also available for these type to pull results. The following list is the commonly used interfaces and types that implement IEnumerable<T>:

· System.Collections.Generic.IEnumerable<T>

o Microsoft.Collections.Immutable.IImmutableQueue<T>

§ Microsoft.Collections.Immutable.ImmutableQueue<T>

o Microsoft.Collections.Immutable.IImmutableStack<T>

§ Microsoft.Collections.Immutable.ImmutableStack<T>

o Microsoft.Collections.Immutable.IOrderedCollection<T>

§ Microsoft.Collections.Immutable.ImmutableList<T>

o System.Collections.Concurrent.IProducerConsumerCollection<T>

§ System.Collections.Concurrent.ConcurrentBag<T>

§ System.Collections.Concurrent.ConcurrentQueue<T>

§ System.Collections.Concurrent.ConcurrentStack<T>

o System.Collections.Concurrent.BlockingCollection<T>

o System.Collections.Generic.ICollection<T>

§ System.Collections.Generic.IDictionary<TKey, TValue>

§ System.Collections.Concurrent.ConcurrentDictionary<TKey, TValue>

§ System.Collections.Generic.Dictionary<TKey, TValue>

§ System.Collections.ObjectModel.ReadOnlyDictionary<TKey, TValue>

§ System.Dynamic.ExpandoObject

§ System.Collections.Generic.IList<T>

§ System.ArraySegment<T>

§ System.Collections.Generic.List<T>

§ System.Collections.ObjectModel.Collection<T>

§ System.Collections.ObjectModel.ObservableCollection<T>

§ System.Collections.ObjectModel.KeyedCollection<TKey, TItem>

§ System.Collections.ObjectModel.ReadOnlyCollection<T>

§ System.Collections.Generic.ISet<T>

§ System.Collections.Generic.HashSet<T>

§ System.Collections.Generic.SortedSet<T>

o System.Collections.Generic.IReadOnlyCollection<T>

§ System.Collections.Generic.IReadOnlyDictionary<TKey, TValue>

§ System.Collections.Generic.Dictionary<TKey, TValue>

§ System.Collections.ObjectModel.ReadOnlyDictionary<TKey, TValue>

§ Microsoft.Collections.Immutable.IImmutableDictionary<TKey, TValue>

§ Microsoft.Collections.Immutable.ImmutableDictionary<TKey, TValue>

§ Microsoft.Collections.Immutable.ImmutableSortedDictionary<TKey, TValue>

§ System.Collections.Generic.Dictionary<TKey, TValue>

§ System.Collections.ObjectModel.ReadOnlyDictionary<TKey, TValue>

§ System.Collections.Generic.IReadOnlyList<T>

§ Microsoft.Collections.Immutable.IImmutableList<T>

§ Microsoft.Collections.Immutable.ImmutableList<T>

§ System.Collections.Generic.List<T>

§ System.Collections.ObjectModel.Collection<T>

§ System.Collections.ObjectModel.ReadOnlyCollection<T>

§ Microsoft.Collections.Immutable.IImmutableSet<T>

§ Microsoft.Collections.Immutable.IImmutableHashSet<T>

§ Microsoft.Collections.Immutable.ImmutableHashSet<T>

§ Microsoft.Collections.Immutable.IImmutableSortedSet<T>

§ Microsoft.Collections.Immutable.ImmutableSortedSet<T>

o System.Collections.Generic.LinkedList<T>

o System.Collections.Generic.Queue<T>

o System.Collections.Generic.SortedList<TKey, TValue>

o System.Collections.Generic.Stack<T>

o System.Linq.IGrouping<TKey, TElement>

o System.Linq.ILookup<TKey, TElement>

§ System.Linq.Lookup<TKey, TElement>

o System.Linq.IOrderedEnumerable<TElement>

o System.Linq.ParallelQuery<TSource>*

§ System.Linq.OrderedParallelQuery<TSource>

o System.Linq.IQueryable<T>*

§ System.Linq.IOrderedQueryable<T>

§ System.Linq.EnumerableQuery<T>

§ System.Data.Linq.ITable<TEntity>

§ System.Data.Linq.Table<TEntity>

§ Microsoft.EntityFrameworkCore.DbSet<TEntity>

o System.String (implements IEnumerable<char>)

o T[] (array of T)

In the above list, LINQ to Objects queries cannot be directly used for ParallelQuery<T>, IQueryable<T>, and their subtypes. As fore mentioned, ParallelQuery<T> is defined for parallel LINQ, and IQueryable<T> is defined for remote LINQ, LINQ query methods are also defined for them, like Where extension method for ParallelQuery<T>, and Where extension method for IQueryable<T>, etc. When calling Where for IQueryable<T>, apparently the call is compiled to the Where extension method for IQueryable<T>, not the Enumerable.Where extension method for IEnumerable<T>. To use LINQ to Objects’ Enumerable.Where method for ParallelQuery<T> and IQueryable<T>, AsEnumerable query method can be used, which is discussed later in this chapter. ParallelQuery<T> is covered in the Parallel LINQ chapters, and IQueryable<T> is covered in the LINQ to Entities chapters.

For history reason, there are some types may only implement IEnumerable, but not implement IEnumerable<T>. A Cast query method is provided to cast non-generic sequence to generic sequence for further LINQ to Objects query, which is discussed later.

C# Functional Programming In-Depth (15) Pattern matching

Mon, 17 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

Pattern matching is a common feature in functional languages. C# 7.0 introduces basic pattern matching in is expression and switch statement, including constant value as pattern and type as pattern, and C# 7.1 supports generics in pattern matching.

Pattern matching with is expression

Since C# 1.0, is expression (instance is Type) can be used to test whether the instance is compatible with the specified type. Since C# 7.0. the type can be followed by an optional pattern variable:

internal static void IsTypePattern(object @object)

if (@object is Uri reference)

reference.AbsolutePath.WriteLine();

if (@object is DateTime value)

value.ToString("o").WriteLine();

}

For reference type pattern, is expression is compiled to type conversion with as operation, and null check for the converted reference. The as operator only works for reference type and nullable value type. So, value type pattern matching is compiled to nullable value type conversion with as operator, and HasValue check for the converted nullable value:

internal static void CompiledIsTypePattern(object @object)

Uri reference = @object as Uri;

if (reference != null)

reference.AbsolutePath.WriteLine();

DateTime? nullableValue = @object as DateTime?;

DateTime value = nullableValue.GetValueOrDefault();

if (nullableValue.HasValue)

value.ToString("o").WriteLine();

}

It is also common to use pattern matching with additional test. The following example first uses pattern match to get string from input, then uses out variable to get TimeSpan value:

internal static void IsWithTest(object @object)

if (@object is string @string&& TimeSpan.TryParse(@string, out TimeSpan timeSpan))

timeSpan.TotalMilliseconds.WriteLine();

}

After compilation, the additional test goes after the null check:

internal static void CompiledIsWithTest(object @object)

string @string = @object as string;

if (@string != null &&TimeSpan.TryParse(@string, out TimeSpan timeSpan))

timeSpan.TotalMilliseconds.WriteLine();

}

Before pattern matching is introduced to is expression, the following test-type-then-convert-type syntax is commonly used:

namespace System

public readonly partial struct DateTime

public override bool Equals(object value)

if (value is DateTime)

return this.InternalTicks == ((DateTime)value).InternalTicks;

return false;

public struct TimeSpan

public override bool Equals(object value)

if (value is TimeSpan)

return this._ticks == ((TimeSpan)value)._ticks;

return false;

}

Here the input object’s type was detected twice. Now with the new syntax, the test and conversion can be merged:

namespace System

public readonly partial struct DateTime

public override bool Equals(object value) =>

value is DateTime dateTime && this.InternalTicks == dateTime.InternalTicks;

public struct TimeSpan

public override bool Equals(object value) =>

value is TimeSpan timeSpan && this._ticks == timeSpan._ticks;

}

C# 7.1 supports generics open type in pattern matching:

internal static void IsWithOpenType<TOpen1, TOpen2, TOpen3, TOpen4>(

IDisposable disposable, TOpen2 open2, TOpen3 open3)

if (disposable is TOpen1 open1)

open1.WriteLine();

if (open2 is FileInfo file)

file.WriteLine();

if (open3 is TOpen4 open4)

open4.WriteLine();

}

When open type is involved, the tested instance is boxed as object. When open type is the type pattern, it is unknow whether the open type is reference type or value type. Regarding as operator can only be used for reference type and nullable value type, so as operator cannot be used for the open type here. So, the pattern matching with open type is compiled to the basic is expression of type test.

internal static void CompiledIsWithOpenType<TOpen1, TOpen2, TOpen3, TOpen4>(

IDisposable disposable, TOpen2 open2, TOpen3 open3)

object disposableObject = (object)disposable;

if (disposableObject is TOpen1)

TOpen1 open1 = (TOpen1)disposableObject;

open1.WriteLine();

object open2Object = (object)open2;

FileInfo file = open2Object as FileInfo;

if (file != null)

file.WriteLine();

object open3Object = (object)open3;

if (open3Object is TOpen4)

TOpen4 open4 = (TOpen4)open3Object;

open4.WriteLine();

}

The var keyword can be used to represent the pattern of any type:

internal static void IsType(object @object)

if (@object is var match)

object.ReferenceEquals(@object, match).WriteLine();

}

Since the var pattern matching always works, it is compiled to constant value true in debug build:

internal static void CompiledIsAnyType(object @object)

object match = @object;

if (true)

object.ReferenceEquals(@object, match).WriteLine();

}

In release build, the above if (true) test is removed.

Since C# 7.0, is expression can also test constant pattern (instance is constant), where constant value and constant expression are supported, including primitive type, enumerations, decimal, string, and null for reference type:

internal static void IsConstantPattern(object @object)

bool test1 = @object is null;

bool test2 = @object is default(int);

bool test3 = @object is DayOfWeek.Saturday - DayOfWeek.Monday;

bool test4 = @object is "https://" + "flickr.com/dixin";

bool test5 = @object is nameof(test5);

}

The is expressions for null test is simply compiled to null check. the other cases are compiled to object.Equals static method calls, where the constant value is the first argument, and the tested instance is the second argument.

internal static void CompiledIsConstantPattern(object @object)

bool test1 = @object == null;

bool test2 = object.Equals(0, @object);

bool test3 = object.Equals(5, @object);

bool test4 = object.Equals("https://flickr.com/dixin", @object);

bool test5 = object.Equals("test5", @object);

}

Internally, object.Equals first does a few checks, then it calls Equals instance method of first argument, which is the constant value. Its implementation can be viewed as:

namespace System

[Serializable]

public class Object

public static bool Equals(object objA, object objB) =>

objA == objB || (objA != null && objB != null && objA.Equals(objB));

// Other members.

}

The early version of C# 7.0 compiles the object.Equals static method call in different way. The tested instance is the first argument, and the constant value is the second argument. As a result, object.Equals then calls the tested instance’s Equals instance method. This is problematic, because the tested instance can be of any type, and it can override Equals instance with arbitrary implementation. In C# 7.0 GA release, this was fixed by having the constant value be the first argument of object.Equals static method call. The non-null constant value must be of primitive type, enumeration, decimal, or string, so a constant’s Equals instance method always has reliable built-in implementation. There is no arbitrary custom equality implementation involved in constant pattern matching.

C# uses default(Type) expression to represent the default value of the specified type since 1.0. Then C# 7.1 introduces default literal expression, which just uses the default keyword to represent a default value, with the type omitted and inferred from context. Since then, the default literal expression is also enabled for constant pattern matching:

internal static void IsConstantPatternWithDefault(object @object)

bool test6 = @object is default; // Cannot be compiled. use default(Type).

}

Shortly, this syntax is disabled after C# 7.2 is released, because it causes confusion with the default case in switch statement, which is discussed later in this chapter. For default value pattern matching, use the traditional default(Type) syntax as demonstrated in the first example.

Pattern matching with switch statement and expression

Before C# 7.0, the switch statement only supports string, integral types (like bool, byte, char, int, long, etc.), and enumeration; and the case label only supports constant value. Since C# 7.0, switch supports any type and case label supports pattern matching for either constant value or type. The additional condition for the pattern matching can be specified with a when clause. The following example tries to convert object to DateTime:

internal static DateTime ToDateTime<TConvertible>(object @object)

where TConvertible : IConvertible

switch (@object)

// Match null reference.

case null:

throw new ArgumentNullException(nameof(@object));

// Match value type.

case DateTime dateTIme:

return dateTIme;

// Match value type with condition.

case long ticks when ticks >= 0:

return new DateTime(ticks);

// Match reference type with condition.

case string @string when DateTime.TryParse(@string, out DateTime dateTime):

return dateTime;

// Match reference type with condition.

case int[] date when date.Length == 3 && date[0] > 0 && date[1]> 0 && date[2] > 0:

return new DateTime(year: date[0], month: date[1], day: date[2]);

// Match generics open type.

case TConvertible convertible:

return convertible.ToDateTime(provider: null);

// Match anything else. Equivalent to default case.

case var _:

throw new ArgumentOutOfRangeException(nameof(@object));

}

The last pattern matches any type pattern, so it works equivalently to the default case of switch statement. The special _ identifier is used to discard the pattern variable. In switch statement, each pattern matching is compiled similarly to is expression:

internal static DateTime CompiledToDateTime<TConvertible>(object @object)

where TConvertible : IConvertible

// case null:

if (@object == null)

throw new ArgumentNullException("object");

// case DateTime dateTIme:

DateTime? nullableDateTime = @object as DateTime?;

DateTime dateTime = nullableDateTime.GetValueOrDefault();

if (nullableDateTime.HasValue)

return dateTime;

// case long ticks:

long? nullableInt64 = @object as long?;

long ticks = nullableInt64.GetValueOrDefault();

if (nullableInt64.HasValue && ticks >= 0L) // when clause.

return new DateTime(ticks);

// case string text:

string @string = @object as string;

if (@string != null && DateTime.TryParse(@string, out DateTime parsedDateTime)) // when clause.

return parsedDateTime;

// case int[] date:

int[] date = @object as int[];

if (date != null && date.Length == 3 && date[0] >= 0 && date[1]> = 0 && date[2] >= 0) // when clause.

return new DateTime(date[0], date[1], date[2]);

// case TConvertible convertible:

object convertibleObject = (object)@object;

if (convertibleObject is TConvertible)

TConvertible convertible = (TConvertible)convertibleObject;

return convertible.ToDateTime(null);

// case var _:

throw new ArgumentOutOfRangeException("object");

}

C# 8.0 introduces switch expression syntacticsal sugar to simplify the switch statement:

internal static DateTime ToDateTime2(object @object) =>

@object switch

null => throw new ArgumentNullException(nameof(@object)),

DateTime dateTIme => dateTIme,

long ticks when ticks >= 0 => new DateTime(ticks),

string @string when DateTime.TryParse(@string, out DateTime dateTime) => dateTime,

int[] date when date.Length == 3 && date[0]> 0 && date[1] > 0 && date[2] > 0 => new DateTime(

year: date[0], month: date[1], day: date[2]),

IConvertible convertible => convertible.ToDateTime(provider: null),

_ => throw new ArgumentOutOfRangeException(nameof(@object))

};

Summary

C# has basic pattern matching features. Type pattern (including generics open type) with pattern variable and constant pattern are supported in is expression and switch statement. The var keyword can be used as the pattern of any type, and _ can be used to discard pattern variable. switch statement also supports when clause for each pattern matching for additional condition.

C# Functional Programming In-Depth (12) Immutability, Anonymous Type, and Tuple

Thu, 13 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

Immutability is an important aspect of functional programming. As discussed in the introduction chapter, imperative/object-oriented programming is usually mutable and stateful, and functional programming encourages immutability without state change. There are many kinds of immutability. In C#, the relevant features can be categorized into 2 levels: immutability of a value, and immutability of a value’s internal state. Take local variable as example, a local variable can be immutable value, if once it is initialized to an instance, it cannot be altered it to different instance to it; a local variable can also have immutable state, if once the instance’s internal state is initialized, that instance’s internal state cannot be altered to different state.

In functional programming, immutability is an important practice. It makes code easier to write, test, and maintain in many cases. Apparently, mutation makes the code unpredictable, and could be a prominent source of bugs; while apparently immutability makes the code predictable, and easier for reasoning about the correctness. Immutable value and immutable state can also make code easier to scale. There is no in-place update, and there is no update conflict. So, immutability is thread-safe by nature, and greatly simplifies concurrent/parallel/multithreading programming.

With immutability, since data cannot be updated in-place, it must be transformed to new data. The disadvantage of immutability is, apparently, the transformation must create another new instance to have the updated state, which can be a performance overhead.

Immutable value

Many functional languages support immutable value. In contrast to variable, once an immutable value is initialized with an instance, it cannot be reassigned so that it can never be altered to another different instance. As a C-like language, C# has variable, which is mutable by default. C# also has a number of language features for immutable value.

Constant local

C# has a const keyword to define compile time constant local. It can be initialized from a constant value or a constant expression that can be evaluated at compiled time, then it cannot be changed at runtime. Constant local only works for primitive types, enumerations, decimal, string, and null for reference type. At compiled time, the reference to constant local is replaced by the actual constant value, and the constant local is removed. The following is a few examples:

internal static void ConstantLocal()

const int ImmutableInt32 = 1;

const int ImmutableInt32Sum = ImmutableInt32 + 2;

// Constant expression ImmutableInt32 + 2 is compiled to: 3.

const DayOfWeek ImmutableDayOfWeek = DayOfWeek.Saturday;

const decimal ImmutableDecimal = (1M + 2M) * 3M;

const string ImmutableString = "https://weblogs.asp.net/dixin";

const string ImmutableStringConcat = "https://" + "flickr.com/dixin";

const Uri ImmutableUri = null;

// Reassignment to above constant locals cannot be compiled.

int variableInt32 = Math.Max(ImmutableInt32, ImmutableInt32Sum);

// Compiled to: Math.Max(1, 3).

Trace.WriteLine(ImmutableString);

// Compiled to: Trace.WriteLine("https://weblogs.asp.net/dixin").

}

Enumeration

Enumeration can be viewed as a group of immutable values of its underlying integral type (int by default). For example:

internal enum Day

Sun, Mon, Tue, Wed, Thu, Fri, Sat

// Compiled to:

internal enum CompiledDay : int

Sun = 0, Mon = 1, Tue = 2, Wed = 3, Thu = 4, Fri = 5, Sat = 6

}

As demonstrated in previous example, enumeration can be used as constant value. It can also be used in constant expression:

internal static void Enumeration()

Trace.WriteLine((int)Day.Mon); // Compiled to: Trace.WriteLine(1).

Trace.WriteLine(Day.Mon + 2); // Compiled to: Trace.WriteLine(Day.Wed).

Trace.WriteLine((int)Day.Mon + Day.Thu); // Compiled to: Trace.WriteLine(Day.Fri).

Trace.WriteLine(Day.Sat - Day.Mon); // Compiled to: Trace.WriteLine(5).

}

using statement and foreach statement

C# also supports immutable value in the fore mentioned using statement and foreach statement:

internal static void Using(Func<IDisposable> disposableFactory)

using (IDisposable immutableDisposable = disposableFactory())

// Reassignment to immutableDisposable cannot be compiled.

internal static void ForEach<T>(IEnumerable<T> sequence)

foreach (T immutableValue in sequence)

// Reassignment to immutableValue cannot be compiled.

}

Immutable alias (immutable ref local variable)

As discussed in the C# basics chapter, ref modifier can define alias for local variable, and readonly modifier can be used with ref modifier to make the alias immutable. Once an immutable alias is initialized with something, it cannot be alias of anything else:

internal static void ImmutableAlias()

int value = 1;

int copyOfValue = value; // Copy.

copyOfValue = 10; // After the assignment, value does not mutate.

ref int aliasOfValue = ref value; // Mutable alias.

aliasOfValue = 10; // After the reassignment, value mutates.

ref readonly int immutableAliasOfValue = ref value; // Immutable alias.

// Reassignment to immutableAliasOfValue cannot be compiled.

Uri reference = new Uri("https://weblogs.asp.net/dixin");

Uri copyOfReference = reference; // Copy.

copyOfReference = new Uri("https://flickr.com/dixin"); // After the assignment, reference does not mutate.

ref Uri aliasOfReference = ref reference; // Mutable alias.

aliasOfReference = new Uri("https://flickr.com/dixin"); // After the reassignment, reference mutates.

ref readonly Uri immutableAliasOfReference = ref reference; // Immutable alias.

// Reassignment to immutableAliasOfReference cannot be compiled.

}

Function’s immutable input and immutable output

As discussed in the function input and output chapter, the in modifier enables function input by immutable alias (in parameter), and the readonly modifier can be used with ref modifier to enable output by immutable alias (ref readonly return):

internal static void ParameterAndReturn<T>(Span<T> span)

ref readonly T First(in Span<T> immutableInput)

// Cannot reassign to immutableInput.

return ref immutableInput[0];

ref readonly T immutableOutput = ref First(in span);

// Cannot reassign to immutableOutput.

}

Range variable in LINQ query expression

The function composition chapter discusses LINQ query expression where, let clause and other clauses can define range variable. LINQ range variable is not mutable variable but immutable value:

internal static void QueryExpression(IEnumerable<int> source1, IEnumerable<int> source2)

IEnumerable<IGrouping<int, int>> query =

from immutable1 in source1

// Reassignment to immutable1 cannot be compiled.

join immutable2 in source2 on immutable1 equals immutable2 into immutable3

// Reassignment to immutable2, immutable3 cannot be compiled.

let immutable4 = immutable1

// Reassignment to immutable4 cannot be compiled.

group immutable4 by immutable4 into immutable5

// Reassignment to immutable5 cannot be compiled.

select immutable5 into immutable6

// Reassignment to immutable6 cannot be compiled.

select immutable6;

}

this reference for class

In class definition, this keyword can be used in instance function members to refer to the current instance of the class, and it is immutable:

internal partial class Device

internal void InstanceMethod()

// Reassignment to this cannot be compiled.

}

For structure, this reference is mutable by default, which is discussed later in this chapter.

Immutable state (immutable type)

A type is immutable type, if once its instance is initialized, the internal state cannot mutate to different state. As a typical example, string (System.String) is an immutable type. Once a string instance is constructed, there is no API to mutate the internal characters of that string instance. For example, string.Remove does not remove specified characters in the string, but constructs a new string without the characters specified to remove. In contrast, string builder (System.Text.StringBuilder) is a mutable type. For example, StringBuilder.Remove mutates the internal characters in place. In .NET Standard, most classes are mutable types, and most structures are immutable types.

Constant field

When defining type (class or structure), a field with the const modifier is static and immutable. It must be initialized by constant value or constant expression inline, and it cannot be reassigned. Similar to constant local, constant field only works for primitive types, enumerations, decimal, string, and null for reference type.

namespace System

public struct DateTime : IComparable, IComparable<DateTime>, IConvertible, IEquatable<DateTime>, IFormattable, ISerializable

private const int DaysPerYear = 365;

// Compiled to:

// .field private static literal int32 DaysPerYear = 365

private const int DaysPer4Years = DaysPerYear * 4 + 1;

// Compiled to:

// .field private static literal int32 DaysPer4Years = 1461

// Other members.

}

At compiled time, the reference to constant field is replaced by the actual constant value, and the constant field definition is not removed.

Immutable class with readonly instance field

When the readonly modifier is used for a field, the field can only be initialized by constructor, and cannot be reassigned later. So an immutable class can be immutable by defining all instance fields as readonly:

internal partial class ImmutableDevice

private readonly string name;

private readonly decimal price;

}

With the fore mentioned auto property syntactic sugar, the readonly field definition can be automatically generated. The following examples are mutable type with read write fields generated from auto property, and immutable type with read only fields generated from getter only auto properties:

internal partial class MutableDevice

internal string Name { get; set; }

internal decimal Price { get; set; }

internal partial class ImmutableDevice

internal ImmutableDevice(string name, decimal price)

this.Name = name;

this.Price = price;

internal string Name { get; }

internal decimal Price { get; }

}

Apparently, to have new internal state, a mutable instance can mutate in place, by altering its fields, while an immutable instance must be transformed to a new instance to have the new state:

internal static void DevicePriceDrop()

MutableDevice mutableDevice = new MutableDevice() { Name = "Surface Laptop", Price = 799M };

mutableDevice.Price -= 50M;

ImmutableDevice immutableDevice = new ImmutableDevice(name: "Surface Book", price: 1199M);

immutableDevice = new ImmutableDevice(name: immutableDevice.Name, price: immutableDevice.Price - 50M);

}

The following examples defines instance method for new state with mutation and transformation:

internal partial class MutableDevice

internal MutableDevice Discount()

this.Price = this.Price * 0.9M;

return this;

internal partial class ImmutableDevice

internal ImmutableDevice Discount() =>

new ImmutableDevice(name: this.Name, price: this.Price * 0.9M);

}

Many .NET built-in types are immutable data structures, including most structures (primitive types, System.Nullable<T>, System.DateTime, System.TimeSpan, etc.), and a few classes (string, System.Lazy<T>, System.Linq.Expressions.Expression and its derived types, etc.). Microsoft also provides a NuGet package of immutable collections System.Collections.Immutable, with immutable array, list, dictionary, etc.

Immutable structure (readonly structure)

The following structure is defined with getter only auto property, which generates readonly fields. The structure looks immutable:

internal partial struct Complex

internal Complex(double real, double imaginary)

this.Real = real;

this.Imaginary = imaginary;

internal double Real { get; }

internal double Imaginary { get; }

}

However, for structure, readonly fields are not enough for immutability. Unlike class, in structure’s instance function members, this reference is mutable:

internal partial struct Complex

internal Complex(Complex value) => this = value; // Can reassign to this.

internal Complex Value

get => this;

set => this = value; // Can reassign to this.

internal Complex ReplaceBy(Complex value) => this = value; // Can reassign to this.

internal Complex Mutate(double real, double imaginary) =>

this = new Complex(real, imaginary); // Can reassign to this.

}

With mutable this, the above structure still can be mutable:

internal static void Structure()

Complex complex = new Complex(1, 1);

complex.Real.WriteLine(); // 1

complex.ReplaceBy(new Complex(2, 2));

complex.Real.WriteLine(); // 2

complex.Mutate(3, 3);

complex.Real.WriteLine(); // 3

}

To address this, C# 7.2 enables the readonly modifier for structure definition to makes sure the structure is immutable. It enforces all the instance fields to be readonly, and makes this reference immutable in instance function members except constructor:

internal readonly partial struct ImmutableComplex

internal ImmutableComplex(double real, double imaginary)

this.Real = real;

this.Imaginary = imaginary;

internal ImmutableComplex(in ImmutableComplex value) =>

this = value; // Can reassign to this only in constructor.

internal double Real { get; }

internal double Imaginary { get; }

internal void InstanceMethod()

// Cannot reassign to this.

}

The readonly modifier is compiled to [IsReadOnly] attribute for the structure:

[IsReadOnly]

internal struct CompiledImmutableComplex

// Members.

}

Immutable anonymous type

C# 3.0 introduces anonymous type to represent temporary data in an immutable form, without providing the type definition at design time:

internal static void AnonymousType()

var immutableDevice = new { Name = "Surface Book", Price = 1199M };

}

Since the type name is unknown at design time, the above instance is of an anonymous type, and the type name is represented by the var keyword. At compile time, the following immutable class is generated:

[CompilerGenerated]

[DebuggerDisplay(@"\{ Name = {Name}, Price = {Price} }", Type = "<Anonymous Type>")]

internal sealed class AnonymousType0<TName, TPrice>

[DebuggerBrowsable(DebuggerBrowsableState.Never)]

private readonly TName name;

[DebuggerBrowsable(DebuggerBrowsableState.Never)]

private readonly TPrice price;

[DebuggerHidden]

public AnonymousType0(TName name, TPrice price)

this.name = name;

this.price = price;

public TName Name => this.name;

public TPrice Price => this.price;

[DebuggerHidden]

[DebuggerHidden]

public override bool Equals(object obj)

AnonymousType0<TName, TPrice> other = obj as AnonymousType0<TName, TPrice>;

return other != null

&&EqualityComparer<TName>.Default.Equals(this.name, other.name)

&&EqualityComparer<TPrice>.Default.Equals(this.price, other.price);

// Other members.

}

The [DebuggerDisplay] attribute is generated for debug build, so that debugger can display the values in the specified friendly format.

And the above setting-property-like syntax is compiled to normal constructor call:

internal static void CompiledAnonymousType()

AnonymousType0<string, decimal>immutableDevice =

new AnonymousType0<string, decimal>(name: "Surface Book", price: 1199M);

}

If there are other different anonymous type used in the code, C# compiler generates more type definitions AnonymousType1, AnonymousType2, etc. Anonymous type is reused by different instantiation if their properties have same number, the same names, the same types, and the same order:

internal static void ReuseAnonymousType()

var device1 = new { Name = "Surface Book", Price = 1199M };

var device2 = new { Name = "Surface Pro", Price = 899M };

var device3 = new { Name = "Xbox One", Price = 399.00 }; // Price is of type double.

var device4 = new { Price = 174.99M, Name = "Surface Laptop" }; // Price is before Name.

(device1.GetType() == device2.GetType()).WriteLine(); // True

(device1.GetType() == device3.GetType()).WriteLine(); // False

(device1.GetType() == device4.GetType()).WriteLine(); // False

}

Anonymous type’s property name can be inferred from the identifier used to initialize the property. The following 2 anonymous type instantiations are equivalent:

internal static void PropertyInference(Uri uri, int value)

var anonymous1 = new { value, uri.Host };

var anonymous2 = new { value = value, Host = uri.Host };

}

Anonymous type can also be part of other types, like array, and type parameter for generic type, etc:

internal static void AnonymousTypeParameter()

var source = // Compiled to: AnonymousType0<string, decimal>[] source =.

new []

new { Name = "Surface Book", Price = 1199M },

new { Name = "Surface Pro", Price = 899M }

};

var query = // Compiled to: IEnumerable<AnonymousType0<string, decimal>> query = .

source.Where(device => device.Price > 0);

}

Here the source array is inferred to be of AnonymousType0<string, decimal>[] type, because each array value is of type AnonymousType0. Array T[] implements IEnumerable<T> interface, so the source array implements IEnumerable<AnonymousType0<string, decimal>> interface. Its Where query method accepts an AnonymousType0<string, decimal> –> bool predicate function, and outputs IEnumerable<AnonymousType0<string, decimal>>.

C# compiler utilizes anonymous type for let clause in LINQ query expression. The let clause is compiled to Select query method call with a selector function. The selector function outputs anonymous type, where each property is a range variable in the context. For example:

internal static void Let(IEnumerable<int> source)

IEnumerable<double> query =

from immutable1 in source

let immutable2 = Math.Sqrt(immutable1)

select immutable1 + immutable2;

internal static void CompiledLet(IEnumerable<int>source)

IEnumerable<double> query = source

.Select(immutable1 => new { immutable1, immutable2 = Math.Sqrt(immutable1) }) // let clause.

.Select(context => context.immutable1 + context.immutable2); // select clause.

}

The full details of query expression compilation are discussed in the LINQ to Objects chapters.

Local variable type inference

Besides local variable of anonymous type, the var keyword can be also used to initialize local variable of existing type:

internal static void LocalVariable(IEnumerable<int> source, string path)

var a = default(int); // int.

var b = 1M; // decimal.

var c = typeof(void); // Type.

var d = from int32 in source where int32 > 0 select Math.Sqrt(int32); // IEnumerable<double>.

var e = File.ReadAllLines(path); // string[].

}

This is just a syntactic sugar. The local variable’s type is inferred from the initial value’s type. The compilation of implicit typed local variable has no difference from explicitly typed local variable. When the initial value’s type is ambiguous, the var keyword cannot be directly used:

internal static void LocalVariableWithType()

var f = (Uri)null;

var g = (Func<int, int>)(int32 => int32 + 1);

var h = (Expression<Func<int, int>>)(int32 => int32 + 1);

}

For consistency and readability, this book uses explicit typing when possible, uses implicit typing (var) when needed.

Immutable tuple and mutable tuple

Tuple is a data structure commonly used in functional programming. It is a finite and ordered list of values, usually immutable in most functional languages. To represent tuple, a series of generic tuple classes with 1 - 8 type parameters are provided in .NET Standard. All tuple classes are immutable. For example, the following is the definition of Tuple<T1, T2>, which represents a 2-tuple (tuple of 2 values):

namespace System

[Serializable]

public class Tuple<T1, T2> : IStructuralEquatable, IStructuralComparable, IComparable, ITupleInternal, ITuple

private readonly T1 m_Item1;

private readonly T2 m_Item2;

public Tuple(T1 item1, T2 item2)

this.m_Item1 = item1;

this.m_Item2 = item2;

public T1 Item1 => this.m_Item1;

public T2 Item2 => this.m_Item2;

// Other members.

}

Years after introducing tuple classes, C# 7.0 finally introduces tuple syntax. However, the tuple syntax does not work with above immutable tuple classes, but works with a new series of generic tuple structures with 1 ~ 8 type parameters. For example, 2-tuple is now represented by the following ValueTuple<T1, T2> structure:

namespace System

[StructLayout(LayoutKind.Auto)]

public struct ValueTuple<T1, T2> : IEquatable<ValueTuple<T1, T2>>, IStructuralEquatable, IStructuralComparable, IComparable, IComparable<ValueTuple<T1, T2>>, IValueTupleInternal, ITuple

public T1 Item1;

public T2 Item2;

public ValueTuple(T1 item1, T2 item2)

this.Item1 = item1;

this.Item2 = item2;

public override string ToString() =>

"(" + this.Item1?.ToString() + ", " + this.Item2?.ToString() + ")";

// Other members.

}

The tuple structures with fields are provided for better performance than tuple classes with properties, since structure can be allocated and deallocated on stack, and field access does not involve getter function member call. Unfortunately, all tuple structures are designed to be mutable types with non-readonly public fields. To be functional and consistent, this book does not use tuple classes, but only uses tuple structures and never mutate their value fields, as if tuple structures are immutable.

As above tuple definition shows, in contrast of list of values or array of values, tuple’s values can be of different types:

internal static void TypesOfValues()

ValueTuple<string, decimal> tuple = new ValueTuple<string, decimal>("Surface Book", 1199M);

string[] array = { "Surface Book", "1199M" };

List<string>list = new List<string>() { "Surface Book", "1199M" };

}

Tuple type and anonymous type are conceptually similar to each other, they are both a list of values of arbitrary types. At design time, tuple type is defined, and anonymous type is not defined yet. Therefore, anonymous type (var keyword) can only be used for local variable with immediate instantiation to infer the types of the values, and cannot be used as function input type, function output type, type argument, etc.:

internal static ValueTuple<string, decimal> Function(ValueTuple<string, decimal> values)

ValueTuple<string, decimal> variable1;

ValueTuple<string, decimal> variable2 = default;

IEnumerable<ValueTuple<string, decimal>>variable3;

return values;

internal static var Function(var values) // Cannot be compiled.

var variable1; // Cannot be compiled.

var variable2 = default; // Cannot be compiled.

IEnumerable<var> variable3; // Cannot be compiled.

return values;

}

Construction, element and element inference

C# 7.0 introduces tuple syntactic sugar, which brings great convenience. The tuple type ValueTuple<T1, T2, T3, …> can be simplified to (T1, T2, T3, …), and the tuple construction new ValueTuple<T1, T2, T3, …>(value1, value2, value3, …) can be simplified to (value1, value2, value3, …):

internal static void TupleLiteral()

(string, decimal) tuple1 = ("Surface Pro", 899M);

// Compiled to:

// ValueTuple<string, decimal> tuple1 = new ValueTuple<string, decimal>("Surface Pro", 899M);

(int, bool, (string, decimal)) tuple2 = (1, true, ("Surface Studio", 2999M));

// ValueTuple<int, bool, ValueTuple<string, decimal>> tuple2 =

// new ValueTuple<int, bool, ValueTuple<string, decimal>>(1, true, new ValueTuple<string, decimal>("Surface Studio", 2999M));

}

When using tuple as the function output type, the tuple syntax virtually enables function to return multiple values:

internal static (string, decimal) OutputMultipleValues()

// Compiled to: internal static ValueTuple<string, decimal> OutputMultipleValues()

string value1 = default;

int value2 = default;

(string, decimal) Function() => (value1, value2);

// Compiled to: ValueTuple<string, decimal> Function() => new ValueTuple<string, decimal>(value1, value2);

Func<(string, decimal)> function = () => (value1, value2);

// Compiled to: Func<ValueTuple<string, decimal>> function = () => new ValueTuple<string, decimal>(value1, value2);

return (value1, value2);

// Compiled to : new ValueTuple<string, decimal>(value1, value2);

}

C# 7.0 also introduces element name, so that each value of the tuple type definition can be given a property-like name, with the syntax (T1 Name1, T2 Name2, T3 Name3, …), and each value of the tuple instance construction can be given a name too, with syntax (Name1: value1, Name2, value2, Name3 value3, …). So that the values in the tuple can be accessed with a meaningful name, instead of the actual Item1, Item2, Item3, … field names. This is also a syntactic sugar, at compile time, all element names are all replaced by the underlying fields.

internal static void ElementName()

(string Name, decimal Price) tuple1 = ("Surface Pro", 899M);

tuple1.Name.WriteLine();

tuple1.Price.WriteLine();

// Compiled to:

// ValueTuple<string, decimal> tuple1 = new ValueTuple<string, decimal>("Surface Pro", 899M);

// TraceExtensions.WriteLine(tuple1.Item1);

// TraceExtensions.WriteLine(tuple1.Item2);

(string Name, decimal Price) tuple2 = (ProductNanme: "Surface Book", ProductPrice: 1199M);

tuple2.Name.WriteLine(); // Element names on the right side are ignored when there are element names on the left side.

var tuple3 = (Name: "Surface Studio", Price: 2999M);

tuple3.Name.WriteLine(); // Element names are available through var.

ValueTuple<string, decimal> tuple4 = (Name: "Xbox One", Price: 179M);

tuple4.Item1.WriteLine(); // Element names are not available on ValueTuple<T1, T2> type.

tuple4.Item2.WriteLine();

(string Name, decimal Price) Function((string Name, decimal Price) tuple)

tuple.Name.WriteLine(); // Input tuple’s element names are available in function.

return (tuple.Name, tuple.Price - 10M);

var tuple5 = Function(("Xbox One S", 299M));

tuple5.Name.WriteLine(); // Output tuple’s element names are available through var.

tuple5.Price.WriteLine();

Func<(string Name, decimal Price), (string Name, decimal Price)> function = tuple =>

tuple.Name.WriteLine(); // Input tuple’s element names are available in function.

return (tuple.Name, tuple.Price - 100M);

};

(string ProductName, decimal ProductPrice) tuple6 = function(("HoloLens", 3000M));

tuple6.ProductName.WriteLine(); // Element names on the right side are ignored when there are element names on the left side.

tuple6.ProductPrice.WriteLine();

}

Similar to anonymous type’s property name inference, C# 7.1 can infer tuple’s element name from the identifier used to initialize the element. The following 2 tuple are equivalent:

internal static void ElementInference(Uri uri, int value)

var tuple1 = (value, uri.Host);

var tuple2 = (value: value, Host: uri.Host);

}

Deconstruction

Since C# 7.0, the var keyword can also be used to deconstruct tuple to a list of values. This syntax is very useful when used with functions that outputs multiple values represented by tuple:

internal static void DeconstructTuple()

(string, decimal) GetProduct() => ("HoloLens", 3000M);

var (name, price) = GetProduct();

name.WriteLine();

price.WriteLine();

}

This deconstruction syntactic sugar can be used with any type, as long as that type has a Deconstruct instance or extension method defined, where the values as the out parameters. Take the fore mentioned Device type as example, it has 3 properties Name, Description, and Price, so its Deconstruct method can be either of the following 2 forms:

internal partial class Device

internal void Deconstruct(out string name, out string description, out decimal price)

name = this.Name;

description = this.Description;

price = this.Price;

internal static class DeviceExtensions

internal static void Deconstruct(this Device device, out string name, out string description, out decimal price)

name = device.Name;

description = device.Description;

price = device.Price;

}

Now the var keyword can destruct Device too, which is just compiled to Destruct method call:

internal static void DeconstructDevice()

Device GetDevice() => new Device() { Name = "Surface studio", Description = "All-in-one PC.", Price = 2999M };

var (name, description, price) = GetDevice();

// Compiled to:

// string name; string description; decimal price;

// surfaceStudio.Deconstruct(out name, out description, out price);

name.WriteLine();

description.WriteLine();

price.WriteLine();

}

Discard

In tuple destruction, since the elements are compiled to out variables of the Destruct method, any element can be discarded with underscore just like discarding out variable:

internal static void Discard()

Device GetDevice() => new Device() { Name = "Surface studio", Description = "All-in-one PC.", Price = 2999M };

var (_, _, price1) = GetDevice();

price1.WriteLine();

(_, _, decimal price2) = GetDevice();

price2.WriteLine();

}

Tuple assignment

With the tuple syntax, now C# can also support fancy tuple assignment, just like in Python and other languages. The following example assigns 2 values to 2 variables with a single line of code, then swap the values of 2 variables with a single line of code:

internal static void TupleAssignment(int value1, int value2)

(value1, value2) = (1, 2);

// Compiled to:

// value1 = 1; value2 = 2;

(value1, value2) = (value2, value1);

// Compiled to:

// int temp1 = value1; int temp2 = value2;

// value1 = temp2; value2 = temp1;

}

It is easy to calculate Fibonacci number with loop and tuple assignment:

internal static int Fibonacci(int n)

(int a, int b) = (0, 1);

for (int i = 0; i< n; i++)

(a, b) = (b, a + b);

return a;

}

Besides variables, tuple assignment works for other scenarios too, like type member. The following example assigns 2 values to 2 properties with a single line of code:

internal class ImmutableDevice

internal ImmutableDevice(string name, decimal price) =>

(this.Name, this.Price) = (name, price);

internal string Name { get; }

internal decimal Price { get; }

}

Immutable collection vs. readonly collection

Microsoft provides immutable collections through the System.Collections.Immutable NuGet Package, including ImmutableArray<T>, ImmutableDictionary<TKey, TValue>, ImmutableHashSet<T>, ImmutableList<T>, ImmutableQueue<T>, ImmutableSet<T>, ImmutableStack<T>, etc. As fore mentioned, trying to mutate an immutable collection creates a new immutable collection:

internal static void ImmutableCollection()

ImmutableList<int> immutableList1 = ImmutableList.Create(1, 2, 3);

ImmutableList<int> immutableList2 = immutableList1.Add(4); // Create a new collection.

object.ReferenceEquals(immutableList1, immutableList2).WriteLine(); // False

}

.NET Standard also provides readonly collections, like ReadOnlyCollection<T>, ReadOnlyDictionary<TKey, TValue>, etc. These readonly collections are actually a simple wrapper of the specified collections. They just do not implement or expose mutation methods like Add, Remove, etc. They are not immutable or thread safe. The following example initializes an immutable collection and a readonly collection from a mutable source. When the source is mutated, the immutable collection apparently is not impacted, but the readonly collection mutates too:

internal static void ReadOnlyCollection()

List<int> mutableList = new List<int>() { 1, 2, 3 };

ImmutableList<int> immutableList = ImmutableList.CreateRange(mutableList);

ReadOnlyCollection<int>readOnlyCollection = new ReadOnlyCollection<int>(mutableList);

mutableList.Add(4);

immutableList.Count.WriteLine(); // 3

readOnlyCollection.Count.WriteLine(); // 4

}

Shallow immutability vs. deep immutability

When working with immutable types, it is noteworthy that how immutable this type is. Take the following type as example, it represents a bundle of 2 devices:

internal class Bundle

internal Bundle(MutableDevice device1, MutableDevice device2) =>

(this.Device1, this.Device2) = (device1, device2);

internal MutableDevice Device1 { get; }

internal MutableDevice Device2 { get; }

}

This type is apparently immutable. Once it is initialized with 2 device instances, there is no way to alter these 2 instances to different instances. However, these 2 device instances are mutable, their names and prices can be altered in place:

internal static void ShallowImmutability()

MutableDevice device1 = new MutableDevice() { Name = "Surface Book", Price = 1199M };

MutableDevice device2 = new MutableDevice() { Name = "HoloLens", Price = 3000M };

Bundle bundle = new Bundle(device1, device2);

// Reassignment to bundle.Device1, bundle.Device2 cannot be compiled.

bundle.Device1.Name = "Surface Studio";

bundle.Device1.Price = 2999M;

bundle.Device2.Price -= 50M;

}

So, the above type is only immutable to a certain level. Below that level, mutation can happen. This is called shallow immutability. In contrast, the following type is more immutable:

internal class Bundle

internal Bundle(ImmutableDevice device1, ImmutableDevice device2) =>

(this.Device1, this.Device2) = (device1, device2);

internal ImmutableDevice Device1 { get; }

internal ImmutableDevice Device2 { get; }

}

This time a bundle consists of 2 immutable devices, and each immutable device consists of an immutable int value and an immutable string value. So, this type is immutable all levels down and completely thread-safe, which is called deep immutability.

Summary

Immutability is important in functional programming. C# has a number of features for immutable value, including constant local, enumeration, using statement and foreach statement, immutable alias, function’s immutable input and immutable output, LINQ expression’s range variable, and this reference in class instance function members. To implement immutable state inside an instance, C# provides constant field and readonly field for type, readonly structure, immutable anonymous type, and tuple. Microsoft also provide a library of immutable collections.

C# Functional Programming In-Depth (13) Pure Function

Thu, 13 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

The previous chapter discusses that functional programming encourages modelling data as immutable. Functional programming also encourages modelling operations as pure functions. The encouraged purity for function is the significant difference from method and procedure in imperative programming.

Pure function and impure function

A function is pure if:

· Its output only depends on input, and does not depend on anything outside the function, like global variable, input from outside world to the hardware, etc. In another word, given the same input, it always gives the same output.

· It does not have obvious interaction with the environment outside the function or the world outside the hardware when it is called. In another word, the function has no side effect. Here are some examples of side effect:

o Mutate state

o Mutate arguments, free variable, or global variable, etc.

o Produce I/O (input/output between the hardware running the function and the outside world of that hardware)

So pure function is like mathematics function, which is a deterministic mapping between a set of input and a set of output. Other functions are called impure functions.

For example, the following functions are not deterministic, and they are impure:

· Console.Read, Console.ReadLine, Console.ReadKey: give unpredictable output when called each time, since the functions’ output totally depends on the input from outside world to the hardware.

· Random.Next, Guid.NewGuid, System.IO.Path.GetRandomFileName: give random output when called each time.

· DateTime.Now, DateTimeOffset.Now: give different output when called at different time.

And the following functions have side effects and they are impure:

· Property setter: usually mutate state.

· System.Threading.Thread.Start, Thread.Abort: mutate thread state and hardware state.

· int.TryParse, Interlocked.Increase: mutate argument through out/ref parameter.

· Console.Read, Console.ReadLine, Console.ReadKey, Console.Write, Console.WriteLine: produce console I/O.

· System.IO.Directory.Create, Directory.Move, Directory.Delete, File.Create, File.Move, File.Delete, File.ReadAllBytes, File.WriteAllBytes: produce file system I/O.

· System.Net.Http.HttpClient.GetAsync, HttpClient.PostAsync, HttpClinet.PutAsync, HttpClient.DeleteAsync: produce network I/O.

· IDisposable.Dispose: interact with environment to release unmanaged resources, and usually mutate the current state to disposed.

Strictly speaking, any function can interact with the outside world. A function call can at least have the hardware consuming electric energy from outside world and producing heat to outside world. When identifying function’s purity, only explicit interactions are considered.

In contrast, the following examples are pure functions because they are both deterministic and side effect free:

· Most mathematics functions, like numeric primitive types and decimal’s arithmetic operators, most of System.Math type’s static methods, etc. Take Math.Max and Math.Min as examples, their computed output only depends on the input, and they do not produce any side effect, like mutating state/argument/global variable, producing I/O, etc.:

namespace System

public static class Math

public static int Max(int val1, int val2) => (val1 >= val2) ? val1 : val2;

public static int Min(int val1, int val2) => (val1 <= val2) ? val1 : val2;

}

· object’s methods, like GetHashCode, GetType, Equals, ReferenceEquals, ToString, etc.

· System.Convert type’ conversion methods, like ToBoolean, ToInt32, etc.

· Property getter to simply output a state without mutating any state or producing other side effect, like string.Length, Nullable<T>.HasValue, Console.Error, etc.

· Immutable type’s function members for transformation, like string.Concat, string.Substring, string.Insert, string.Replace, string.Trim, string.ToUpper, string.ToLower, etc.

Since pure function is deterministic and side effect free, a pure function call expression and a constant expression of the result can always replace each other. This is called referential transparency.

Similar to immutable data, pure function can make code easier to write, test, and maintain in many cases. Pure function does not involve state mutation, and brings all benefits of immutability. Pure function is self-contained without external dependency or interaction, which greatly improves testability and maintainability. If 2 pure function calls have no input/output dependency, the function calls’ order does not matter, which greatly simplifies parallel computing. For example, purity is important in Parallel LINQ query, which is discussed in Parallel LINQ chapters.

As mentioned in the introduction chapter, there is a specialized functional programming paradigm called purely functional programming, which only allows immutable data and pure function. A few languages, like Haskell, are designed for this paradigm only, and they are called purely functional language. Other functional languages, like C#, F#, allows immutable data and mutable data, pure function and impure function, and they are called impure functional language. In purely functional programming, side effects can be managed by monad, like I/O monad, etc. Monad is discussed in the category theory chapters.

Purity in .NET

.NET Standard provides System.Diagnostics.Contracts.PureAttribute. It can be used for a function member to specify that function is pure, or be used for a type to specify all function members of that type are pure:

[Pure]

internal static bool IsPositive(int int32) => int32 > 0;

internal static bool IsNegative(int int32) // Impure.

Console.WriteLine(int32); // Side effect: console I/O.

return int32 < 0;

[Pure]

internal static class AllFunctionsArePure

internal static int Increase(int int32) => int32 + 1; // Pure.

internal static int Decrease(int int32) => int32 - 1; // Pure.

}

Looks great. Unfortunately, this attribute is provided not for general purpose but only for Code Contracts, a .NET tool provided (and now discontinued) by Microsoft. Code Contracts tool consists of:

· Code contracts APIs under System.Diagnostics.Contracts namespace to specify preconditions, post conditions, invariant, purity, etc., including the above PureAttribute.

· Contracts assemblies for most commonly used FCL assemblies

· Compile time rewriter and analyzer

· Runtime analyzer

In a function member, the contracts for its code can be specified declaratively with System.Diagnostics.Contracts.Contract type’s static methods. These contracts can be analysed at compile time and runtime. Apparently, they must be referential transparent and cannot rely on any side effect. So, only pure function (function with [Pure] contract) are allowed to be called with contracts APIs, like the above IsPositive function:

internal static int DoubleWithPureContracts(int int32)

Contract.Requires<ArgumentOutOfRangeException>(IsPositive(int32)); // Function precondition.

Contract.Ensures(IsPositive(Contract.Result<int>())); // Function post condition.

return int32 + int32; // Function body.

}

In contrast, the following example calls impure function (function without [Pure] contract) in contracts:

internal static int DoubleWithImpureContracts(int int32)

Contract.Requires<ArgumentOutOfRangeException>(IsNegative(int32)); // Function precondition.

Contract.Ensures(IsNegative(Contract.Result<int>())); // Function post condition.

return int32 + int32; // Function body.

}

At compile time, Code Contracts gives a warning: Detected call to method IsNegative(System.Int32)' without [Pure] in contracts of method ‘DoubleWithImpureContracts(System.Int32)'.

Code Contracts has been a very useful code tool for compile time and runtime, but Microsoft has discontinued this tool, so it only works for .NET Framework 4.0 and 4.5 on Windows. As a result, [Pure] attribute is actually obsolete.

When code is compiled and built to assembly, its contracts can either be compiled to the same assembly, or to a separate contracts assembly. Since .NET Framework FCL assemblies are already shipped, Microsoft provides 25 contracts assemblies separately for 25 most commonly used assemblies, including mscorlib.Contracts.dll (contracts for mscorlib.dll core library), System.Core.Contracts.dll (contracts for System.Core.dll assembly of LINQ to Objects and LINQ to Parallel, and remote LINQ APIs), System.Xml.Linq.Contracts.dll (contracts for System.Xml.Linq.dll assembly of LINQ to XML), etc. For example, Math.Abs function is provided in mscorlib.dll, so its contracts are provided in mscorlib.Contracts.dll as empty function with the same signature, with only contracts:

namespace System

public static class Math

[Pure]

public static int Abs(int value)

Contract.Requires(value != int.MinValue);

Contract.Ensures(Contract.Result<int>() >= 0);

Contract.Ensures((value - Contract.Result<int>()) <= 0);

return default;

}

C# and .NET Standard are designed in impure paradigm to allow immutability and mutability, purity and impurity. As a result, only a small percentage of the provided functions are pure. This can be demonstrated by utilizing above contracts assemblies and [Pure] contract. The following example has 2 LINQ to Objects queries. The first query counts all pure public functions in the contract assemblies. As fore mentioned, if a function has [Pure] attribute, it is pure; if a type has [Pure] attribute, its functions are all pure. Then the second query counts all public functions in corresponding library assemblies. In both queries, Mono’s reflection library, Mono.Cecil NuGet package, is used, because it can load .NET Framework assemblies and contracts assemblies correctly from different platforms, including Linux/Mac/Windows.

internal internal static void FunctionCount(

string contractsAssemblyDirectory, string assemblyDirectory)

bool HasPureAttribute(ICustomAttributeProvider member) =>

member.CustomAttributes.Any(attribute =>

attribute.AttributeType.FullName.Equals(typeof(PureAttribute).FullName, StringComparison.Ordinal));

string[] contractsAssemblyPaths = Directory

.EnumerateFiles(contractsAssemblyDirectory, "*.Contracts.dll")

.ToArray();

// Query the count of pure functions in all contracts assemblies, including all public functions in public type with [Pure], and all public function members with [Pure] in public types.

int pureFunctionCount = contractsAssemblyPaths

.Select(AssemblyDefinition.ReadAssembly)

.SelectMany(contractsAssembly => contractsAssembly.Modules)

.SelectMany(contractsModule => contractsModule.GetTypes())

.Where(contractsType => contractsType.IsPublic)

.SelectMany(contractsType => HasPureAttribute(contractsType)

? contractsType.Methods.Where(contractsFunction => contractsFunction.IsPublic)

: contractsType.Methods.Where(contractsFunction =>

contractsFunction.IsPublic && HasPureAttribute(contractsFunction)))

.Count();

pureFunctionCount.WriteLine(); // 2223

// Query the count of all public functions in public types in all FCL assemblies.

int functionCount = contractsAssemblyPaths

.Select(contractsAssemblyPath => Path.Combine(

assemblyDirectory,

Path.ChangeExtension(Path.GetFileNameWithoutExtension(contractsAssemblyPath), "dll")))

.Select(AssemblyDefinition.ReadAssembly)

.SelectMany(assembly => assembly.Modules)

.SelectMany(module => module.GetTypes())

.Where(type => type.IsPublic)

.SelectMany(type => type.Methods)

.Count(function => function.IsPublic);

functionCount.WriteLine(); // 82566

}

As a result, in the 25 most commonly used FCL assemblies, there are only 2.69% public function members are pure.

Purity in LINQ

All built-in LINQ query methods have 3 kinds of output: a queryable source, a collection, and a single value. All query methods with queryable source output type are pure functions, the other query methods are impure. For example, the fore mentioned Where, Select, OrderBy query methods of local and remote LINQ are pure:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> predicate);

public static IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector);

public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

// Other members.

public static class Queryable

public static IQueryable<TSource> Where<TSource>(

this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate);

public static IQueryable<TResult> Select<TSource, TResult>(

this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector);

public static IOrderedQueryable<TSource> ThenBy<TSource, TKey>(

this IOrderedQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector);

// Other members.

}

The following ToArray, ToList query methods of local LINQ output collection, and they are impure:

namespace System.Linq

public static class Enumerable

public static TSource[] ToArray<TSource>(this IEnumerable<TSource> source);

public static List<TSource> ToList<TSource>(this IEnumerable<TSource> source);

}

The following First query methods of local and remote LINQ output a single value, and they are also impure:

namespace System.Linq

public static class Enumerable

public static TSource First<TSource>(this IEnumerable<TSource> source);

public static class Queryable

public static TSource First<TSource>(this IQueryable<TSource> source);

}

The query methods with queryable source output are implemented as pure function by simply constructing a new source instance with the input source. Calling these query methods only deterministically defines a new query and the query execution is deferred, so there is no state mutation or side effect. The output new query can be executed by pulling the results, which can mutate the query’s state and can produce side effect. The other query methods with collection or single value output are different. They immediately execute the pulling from their input source, which can mutate state and can produce side effect, so they are impure. The internal implementation of these methods is covered in the LINQ chapters.

Summary

Pure function is important in functional programming. Pure function is deterministic and side effect free, so they are easy to write, test, maintain and parallelize. In LINQ, all query methods with queryable source output are pure functions.

C# Functional Programming In-Depth (11) Covariance and Contravariance

Tue, 11 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

In programming languages with subtyping support, variance is the ability to substitute a type with a different subtype or supertype in a context. For example, having a subtype and a supertype, can a function of subtype output substitute a function of supertype output? Can a sequence of subtype substitute a sequence of supertype? C# supports covariance/contravariance, which means to use a more/less specific type to substitute the required type in the context of function and interface.

Subtyping and type polymorphism

The following is a simple example of subtyping:

internal interface ISupertype { /* Members. */ }

internal interface ISubtype : ISupertype { /* More members. */ }

Here ISubtype interface implements ISupertype interface. In this relationship, ISubtype is more specific supertype, and ISupertype is less specific supertype, which can be denoted ISubtype <: ISupertype. A subtype instance can substitute a supertype instance. In another word, an instance’s type can be more specific than required type:

internal static void Substitute(ISupertype supertype, ISubtype subtype)

supertype = subtype;

}

In object-oriented programming, subtyping is intuitive with inheritance:

internal class Base { }

internal class Derived : Base { }

Similarly, Derived is a more derived and more specific subtype, and Base is a less derived and less specific supertype. Apparently Derived <: Base (called a Derived instance “is a” Base instance in object-oriented programming), and Derived instance can substitute Base instance (called Liskov substitution principle in object-oriented programming):

internal static void Substitute()

Base @base = new Base();

@base = new Derived();

}

C#’s covariance and contravariance enables the subtyping and substitution relationship for functions and generic interfaces. Subtyping does not have to be inheritance, but to be intuitive to most C# developers, this chapter uses above Base and Derived types to demonstrate variances in all contexts. As discussed in the C# basics chapter, in C#, value types consist of structures and enumerations. All structures inherit System.ValueType class, and all enumerations inherit System.Enum class, so one value type cannot be subtype of another value type. As a result, C#’s covariance and contravariance only applies to reference types, so the above Base and Derived types are defined as classes.

Variances of non-generic function type

When using above Base and Derived as input and output type of function, there are 4 cases:

// Derived -> Base

internal static Base DerivedToBase(Derived input) => new Base();

// Derived -> Derived

internal static Derived DerivedToDerived(Derived input) => new Derived();

// Base -> Base

internal static Base BaseToBase(Base input) => new Base();

// Base -> Derived

internal static Derived BaseToDerived(Base input) => new Derived();

Their function types can be represented by the following non-generic delegate types:

// Derived -> Base

internal delegate Base DerivedToBase(Derived input);

// Derived -> Derived

internal delegate Derived DerivedToDerived(Derived input);

// Base -> Base

internal delegate Base BaseToBase(Base input);

// Base -> Derived

internal delegate Derived BaseToDerived(Base input);

Take the first function DerivedToBase as example, it is of type Derived -> Base, represented by the first delegate type DerivedToBase:

internal static void NonGenericDelegate()

DerivedToDerived derivedToDerived = DerivedToDerived;

Derived output = derivedToDerived(input: new Derived());

}

The second function DerivedToDerived is not of type Derived -> Base, but Derived -> Derived, represented by the second delegate type. Since C# 2.0, DerivedToBase can be substituted by DerivedToDerived in context of non-generic delegate:

internal static void NonGenericDelegateCovariance()

DerivedToBase derivedToBase = DerivedToBase; // Derived -> Base

// Covariance: Derived <: Base, so that Derived -> Derived <: Derived -> Base.

derivedToBase = DerivedToDerived; // Derived -> Derived

// When calling derivedToBase, DerivedToDerived executes.

// derivedToBase should output Base, while DerivedToDerived outputs Derived.

// The actual Derived output substitutes the required Base output. This call always works.

Base output = derivedToBase(input: new Derived());

}

So, function with more derived/specific output can substitute function with less derived/specific output, as if former function type is subtype and latter function type is supertype. In another word, function instance’s actual output can be more derived/specific than function type’s required output. This is called covariance, because it persists the direction of subtyping/substitution: if Derived <: Based then function with Derived output <: function with Base output. Similarly, function instance’s input can be less derived/specific than function type input:

internal static void NonGenericDelegateContravariance()

DerivedToBase derivedToBase = DerivedToBase; // Derived -> Base

// Contravariance: Derived <: Base, so that Base -> Base <: Derived ->Base.

derivedToBase = BaseToBase; // Base -> Base

// When calling derivedToBase, BaseToBase executes.

// derivedToBase should accept Derived input, while BaseToBase accepts Base input.

// The required Derived input substitutes the accepted Base input. This always works.

Base output = derivedToBase(input: new Derived());

}

So, function with less derived/specific input can substitute function with more derived input, or in another word, function instance’s output can be less derived/specific than function type’s required input. This is called contravariance, because it inverts the direction of subtyping/substitution: if Derived <: Base then function with Base input< : function with Derived inp_ut._ Covariance (for output) and contravariance (for input) can work at the same time:

internal static void NonGenericDelegateCovarianceAndContravariance()

DerivedToBase derivedToBase = DerivedToBase; // Derived -> Base

// Covariance and contravariance: Derived <: Base, so that Base -> Derived <: Derived -> Base.

derivedToBase = BaseToDerived; // Base -> Derived

// When calling derivedToBase, BaseToDerived executes.

// derivedToBase should accept Derived input, while BaseToDerived accepts Base input.

// The required Derived input substitutes the accepted Base input.

// derivedToBase should output Base, while BaseToDerived outputs Derived.

// The actual Derived output substitutes the required Base output. This always works.

Base output = derivedToBase(input: new Derived());

}

On the other hand, function output cannot be less derived/specific than required, and function input cannot be more derived/specific than required. The following function substitution cannot be compiled:

internal static void NonGenericDelegateInvariance()

// baseToDerived should output Derived, while BaseToBase outputs Base.

// The actual Base output does not substitute the required Derived output. This cannot be compiled.

BaseToDerived baseToDerived = BaseToBase; // Base -> Derived

// baseToDerived should accept Base input, while DerivedToDerived accepts Derived input.

// The required Base input does not substitute the accepted Derived input. This cannot be compiled.

baseToDerived = DerivedToDerived; // Derived -> Derived

// baseToDerived should accept Base input, while DerivedToBase accepts Derived input.

// The required Base input does not substitute the expected Derived input.

// baseToDerived should output Derived, while DerivedToBase outputs Base.

// The actual Base output does not substitute the required Derived output. This cannot be compiled.

baseToDerived = DerivedToBase; // Derived -> Base

}

Variances of generic function type

With generic delegate type, all the above function types can be represented by:

internal delegate TOutput GenericFunc<TInput, TOutput>(TInput input);

Then the delegate instance can be initialized with variances:

internal static void GenericDelegateCovarianceAndContravariance()

GenericFunc<Derived, Base> derivedToBase = DerivedToBase; // No variance.

derivedToBase = DerivedToDerived; // Covariance.

derivedToBase = BaseToBase; // Contravariance.

derivedToBase = BaseToDerived; // Covariance and contravariance.

}

For functions of GenericFunc<TInput, TOutput> type, covariance enables TOutput to be substituted by more specific type, and contravariance enables TInput to be substituted by less derived type. So TOutput/TInput are called covariant/contravariant type parameter for this generic delegate type. C# 4.0 introduces the out/in modifiers for the covariant/contravariant type parameter:

internal delegate TOutput GenericFuncWithVariances<in TInput, out TOutput>(TInput input);

These modifiers enable further variances, the substitution of generic delegate instances:

internal static void GenericDelegateInstanceSubstitution()

GenericFuncWithVariances<Derived, Base>derivedToBase = DerivedToBase; // Derived -> Base

GenericFuncWithVariances<Derived, Derived>derivedToDerived = DerivedToDerived; // Derived ->Derived

GenericFuncWithVariances<Base, Base>baseToBase = BaseToBase; // Base -> Base

GenericFuncWithVariances<Base, Derived>baseToDerived = BaseToDerived; // Base -> Derived

// Cannot be compiled without the out/in modifiers.

derivedToBase = derivedToDerived; // Covariance.

derivedToBase = baseToBase; // Contravariance.

derivedToBase = baseToDerived; // Covariance and contravariance.

}

As discussed, TInput cannot be covariant, and TOutput cannot be contravariant, in the following definition, TInput with out modifier or TOutput with in modifier cannot work:

// Cannot be compiled.

internal delegate TOutput GenericFuncWithVariances<out TInput, in TOutput>(TInput input);

As mentioned in the delegate chapter, unified Func and Action generic delegate types are provided to represent all function types. Since .NET Framework 4.0, all their type parameters have the out/in modifiers:

namespace System

public delegate TResult Func<out TResult>();

public delegate TResult Func<in T, out TResult>(T arg);

public delegate TResult Func<in T1, in T2, out TResult>(T1 arg1, T2 arg2);

// ...

public delegate void Action();

public delegate void Action<in T>(T obj);

public delegate void Action<in T1, in T2>(T1 arg1, T2 arg2);

// ...

}

Variant type parameter is not syntactic sugar, but a runtime feature. The out/in modifiers are compiled to the +/- flags in CIL, which can be read as can be more/less specific:

.class public auto ansi sealed Func<-T, +TResult> extends System.MulticastDelegate

}

Variances of generic interface

Besides generic function types, C# 4.0 also supports variances for generic interfaces. An interface can be viewed as a group of function types of named function members. For example:

internal interface IOutput<out TOutput> // TOutput is covariant for all members using TOutput.

TOutput ToOutput(); // TOutput is covariant.

TOutput Output { get; } // Compiled to get_Output. TOutput is covariant.

void TypeParameterNotUsed();

}

The above generic interface definition has 3 function members, where 2 function members uses the type parameter only as output type. Apparently, the type parameter is covariant for both 2 functions’ function types. In this case, the type parameter is covariant for the interface level, and the out modifier can be used to enable the substitution of interface instances:

internal static void GenericInterfaceCovariance(

IOutput<Base> outputBase, IOutput<Derived> outputDerived)

// Covariance: Derived <: Base, so that IOutput<Derived> <: IOutput<Base>.

outputBase = outputDerived;

// When calling outputBase.ToOutput, outputDerived.ToOutput executes.

// outputBase.ToOutput should output Base, outputDerived.ToOutput outputs Derived.

// The actual Derived output substitutes the required Base output. This always works.

Base output1 = outputBase.ToOutput();

Base output2 = outputBase.Output; // .get_Output();

}

IOutput<Derived> interface does not implement IOutput<Base> interface, but with out modifier, IOutput<Derived> instance can still substitute IOutput<Base> instance, as if IOutput<Derived> is subtype and IOutput<Base> is supertype.

Similarly, generic interface can also have contravariant type parameter, with in modifier:

internal interface IInput<in TInput> // TInput is contravariant for all members using TInput.

void InputToVoid(TInput input); // TInput is contravariant.

TInput Input { set; } // Compiled to set_Input. TInput is contravariant.

void TypeParameterNotUsed();

}

The above generic interface definition has 3 function members, where 2 function members uses the type parameter only as input type. Apparently, the type parameter is contravariant for both 2 functions’ function types. In this case, the type parameter is contravariant for the interface level, and the in modifier can be used to enable the substitution of interface instances:

internal static void GenericInterfaceContravariance(

IInput<Derived> inputDerived, IInput<Base> inputBase)

// Contravariance: Derived <: Base, so that IInput<Base> <: IInput<Derived>.

inputDerived = inputBase;

// When calling inputDerived.Input, inputBase.Input executes.

// inputDerived.Input should accept Derived input, while inputBase.Input accepts Base input.

// The required Derived output substitutes the accepted Base input. This always works.

inputDerived.InputToVoid(input: new Derived());

inputDerived.Input = new Derived(); // .set_Input(input: new Derived());

}

IInput<Base> interface does not implement IInput<Derived> interface, but with in modifier, IInput<Base> instance can still substitute IInput<Derived> instance, as if IInput<Base> is subtype and IInput<Derived> is supertype.

Similar to generic delegate type, generic interface can have covariant type parameter and contravariant type parameter at the same time:

internal interface IInputOutput<in TInput, out TOutput> // TInput is contravariant for all members using TInput, TOutput is covariant for all members usingTOutput.

void InputToVoid(TInput input); // TInput is contravariant.

TInput Input { set; } // Compiled to set_Input. TInput is contravariant.

TOutput ToOutput(); // TOutput is covariant.

TOutput Output { get; } // Compiled to get_Output. TOutput is covariant.

void TypeParameterNotUsed();

}

The following example demonstrates the covariance and contravariance:

internal static void GenericInterfaceCovarianceAndContravariance(

IInputOutput<Derived, Base>inputDerivedOutputBase,

IInputOutput<Base, Derived> inputBaseOutputDerived)

// Covariance and contravariance: Derived <: Base, so that IInputOutput<Base, Derived> <: IInputOutput<Derived, Base>.

inputDerivedOutputBase = inputBaseOutputDerived;

inputDerivedOutputBase.InputToVoid(new Derived());

inputDerivedOutputBase.Input = new Derived(); // .set_Input(input: new Derived());

Base output1 = inputDerivedOutputBase.ToOutput();

Base output2 = inputDerivedOutputBase.Output; // .get_Output();

}

Not all type parameters can be variant for generic interface. For example:

internal interface IInvariant<T>

T Output(); // T is covariant.

void Input(T input); // T is contravariant.

}

The type parameter T is neither covariant for all function members using T, nor contravariant for all function members using T, so T cannot be covariant or contravariant for the interface level.

Variances of generic higher-order function type

The variances are interesting for generic higher-order function types. Higher-order function type can be defined by outputting function, since it outputs a function. Regarding the type parameter is only used as output, use the out modifier:

// () -> () -> TOutput Equivalent to Func<Func<TOutput>>

internal delegate Func<TOutput> ToFunc<out TOutput>(); // Covariant output type.

Marking the type parameter as covariant, the code can still be compiled. The following example demonstrates how the variance and substitution work at runtime:

internal static void OutputCovariance()

// First order functions.

Func<Base>toBase = () => new Base(); // () -> Base

Func<Derived> toDerived = () => new Derived(); // () -> Derived

// Higher-order functions.

ToFunc<Base>toToBase = () => toBase; // () -> () -> Base

ToFunc<Derived> toToDerived = () => toDerived; // () -> () -> Derived

// Covariance: Derived <: Base, so that () -> () -> Derived <: () -> () -> Base.

toToBase = toToDerived;

// When calling toToBase, toToDerived executes.

// toToBase should output Func<Base>, while toToDerived outputs Func<Derived>.

// The actual Func<Derived> output substitutes the required Func<Base> output. This always works.

Func<Base>output = toToBase();

}

The reason is, covariance persists the direction of subtyping and substitution, Derived <: Base in Context<out T> leads to Context<Derived> <: Context<Base>, Context<Context<Derived>> <: Context<Context<Base>>, Context<Context<Context<Derived>>> <: Context<Context<Context<Derived>>>, and so on. T is always covariant for Context<Context<T>>, Context<Context<Context<T>>>, etc. Take above Func<out TResult> as example:

// () -> TOutput

internal delegate TOutput Func<out TOutput>(); // Covariant.

// () -> () -> TOutput: Equivalent to Func<Func<TOutput>>.

internal delegate Func<TOutput> ToFunc<out TOutput>(); // Covariant.

// () -> () -> () -> TOutput: Equivalent to Func<Func<Func<TOutput>>>.

internal delegate ToFunc<TOutput> ToToFunc<out TOutput>(); // Covariant.

// () -> () -> () -> () -> TOutput: Equivalent to Func<Func<Func<Func<TOutput>>>>.

internal delegate ToToFunc<TOutput> ToToToFunc<out TOutput>(); // Covariant.

// ...

Higher-order function type can also be defined by accepting function as input. Regarding the type parameter is only used as input, use the in modifier:

// (TInput -> void) -> void: Equivalent to Action<Action<TInput>>.

internal delegate void ActionToVoid<in TTInput>(Action<TTInput> action); // Cannot be compiled.

However, the above code cannot be compiled. Marking the type parameter as covariant works:

// (TInput -> void) -> void: Equivalent to Action<Action<TInput>>.

internal delegate void ActionToVoid<out TInput>(Action<TInput> action);

And this is how the variance and substitution works at runtime:

internal static void InputCovarianceAndContravariance()

// Higher-order functions.

ActionToVoid<Derived> derivedToVoidToVoid = (Action<Derived> derivedToVoid) => { };

ActionToVoid<Base> baseToVoidToVoid = (Action<Base>baseToVoid) => { };

// Covariance: Derived <: Base, so that(Derived -> void) -> void <: (Base -> void) -> void.

baseToVoidToVoid = derivedToVoidToVoid;

// When calling baseToVoidToVoid, derivedToVoidToVoid executes.

// baseToVoidToVoid should accept Action<Base> input, while derivedToVoidToVoid accepts Action<Derived> input.

// The required Action<Derived> input substitutes the accepted Action<Base> input. This always works.

baseToVoidToVoid(default(Action<Base>));

}

The reason is, contravariance inverts the direction of subtyping and substitution, Derived <: Base in Context<in T> leads to Context<Base> <: Context<Derived>, Context<Context<Derived>> <: Context<Context<Base>>, Context<Context<Context<Base>>> <: Context<Context<Context<Derived>>>, and so on. T becomes covariant for Context<Context<T>>, contravariant for Context< Context<Context<T>>>, covariant for Context<Context<Context<Context<T>>>>, etc. Take above Action<in T> as example:

// TInput -> void

internal delegate void Action<in TInput>(TInput input); // Contravariant.

// (TInput -> void) -> void: Equivalent to Action<Action<TInput>>.

internal delegate void ActionToVoid<out TTInput>(Action<TTInput> action); // Covariant.

// ((TInput -> void) -> void) -> void: Equivalent to Action<Action<Action<TInput>>>.

internal delegate void ActionToVoidToVoid<in TTInput>(ActionToVoid<TTInput> actionToVoid); // Contravariant.

// (((TInput -> void) -> void) -> void) -> void: Equivalent to Action<Action<Action<Action<TInput>>>>.

internal delegate void ActionToVoidToVoidToVoid<out TTInput>(ActionToVoidToVoid<TTInput> actionToVoidToVoid); // Covariant.

// ...

Covariance of array

Array T[] implements IList<T> interface:

namespace System.Collections.Generic

public interface IList<T>: ICollection<T>, IEnumerable<T>, IEnumerable

T this[int index] { get; set; }

// Indexer getter is compiled to get_Item. T is covariant.

// Indexer setter is compiled to set_Item. T is contravariant.

// Other members.

}

For IList<T>, T is used by indexer getter and setter function members. T is the output type of the compiled get_Item function, and is the input type of the compiled set_Item function. Apparently, T is neither covariant for both functions’ types, and nor contravariant for both functions’ types. So, T should be invariant for IList<T> and array T[]. However, C# compiler and .NET runtime unexpectedly support covariance for array. The following example can be compiled, but throws ArrayTypeMismatchException at runtime, which can be a source of bugs:

internal static void ArrayCovariance()

Base[] baseArray = new Base[3];

Derived[] derivedArray = new Derived[3];

baseArray = derivedArray; // Array covariance: Derived <: Base, so that Derived[]< : Base[], baseArray refers to a Derived array at runtime.

baseArray[1] = new Derived(); // .set_Item(new Derived());

baseArray[2] = new Base(); // .set_Item(new Base());

// ArrayTypeMismatchException at runtime. The actual Derived array requires Derived instance, the provided Base instance cannot substitute Derived instance.

}

Array covariance is first introduced to Java by its designers. They intended to remove this feature from Java around 1995, but they were unable to make it. Later, when .NET Framework is initially released, array covariance is implemented by CLR and C# as a Java feature for the convenience of Java developers. This is a C# language feature that should not be used.

Variances in .NET and LINQ

The following LINQ query finds the generic delegate types and interfaces with variant type parameters in .NET core library:

internal static void TypesWithVariance()

Assembly coreLibrary = typeof(object).Assembly;

coreLibrary.ExportedTypes

.Where(type => type.GetGenericArguments().Any(typeArgument =>

GenericParameterAttributes attributes = typeArgument.GenericParameterAttributes;

return attributes.HasFlag(GenericParameterAttributes.Covariant)

|| attributes.HasFlag(GenericParameterAttributes.Contravariant);

}))

.OrderBy(type => type.FullName)

.WriteLines();

// System.Action`1[T]

// System.Action`2[T1,T2]

// …

// System.Action`8[T1,T2,T3,T4,T5,T6,T7,T8]

// System.Collections.Generic.IComparer`1[T]

// System.Collections.Generic.IEnumerable`1[T]

// System.Collections.Generic.IEnumerator`1[T]

// System.Collections.Generic.IEqualityComparer`1[T]

// System.Collections.Generic.IReadOnlyCollection`1[T]

// System.Collections.Generic.IReadOnlyList`1[T]

// System.Comparison`1[T]

// System.Converter`2[TInput,TOutput]

// System.Func`1[TResult]

// System.Func`2[T,TResult]

// …

// System.Func`9[T1,T2,T3,T4,T5,T6,T7,T8,TResult]

// System.IComparable`1[T]

// System.IObservable`1[T]

// System.IObserver`1[T]

// System.IProgress`1[T]

// System.Predicate`1[T]

}

Under System.Linq namespace, there are also a number of generic interfaces with variance: IGrouping<out TKey, out TElement>, IQueryable<out T>, IOrderedQueryable<out T>. Microsoft has documented the List of Variant Generic Interface and Delegate Types: https://docs.microsoft.com/en-us/dotnet/standard/generics/covariance-and-contravariance#VariantList, but it is inaccurate. It says TElement is covariant for IOrderedEnumerable<TElement>, but actually not:

namespace System.Linq

public interface IOrderedEnumerable<TElement> : IEnumerable<TElement>, IEnumerable

IOrderedEnumerable<TElement> CreateOrderedEnumerable<TKey>(

Func<TElement, TKey> keySelector, IComparer<TKey> comparer, bool descending);

}

Local LINQ query is represented by IEnumerable<T> generic interface, where T is covariant:

namespace System.Collections.Generic

public interface IEnumerator<out T> : IDisposable, IEnumerator

T Current { get; } // Compiled to get_Current.T is covariant.

public interface IEnumerable<out T> : IEnumerable

IEnumerator<T> GetEnumerator();

}

First, In IEnumerator<T> interface, its type parameter is only used as output type of its Current property’s getter, which is compiled to get_Current function. Apparently T is covariance for its function type, so that T is also covariant for IEnumerator<T> interface level. Then, in IEnumerable<T>, T is only used by GetEnumerator. Here IEnumerator<T> can be virtually viewed as a simple wrapper of get_Current function, and GetEnumerator can be virtually viewed as a higher-order function that outputs (a wrapper of) get_Current function. As discussed, T is covariant for get_Current function, covariant for higher-order function GetEnumerator, and eventually covariant for IEnumerable<T> interface level.

Remote LINQ query is represented by IQueryable<T>, where T is also covariant:

namespace System.Linq

public interface IQueryable : IEnumerable

Type ElementType { get; }

Expression Expression { get; }

IQueryProvider Provider { get; }

public interface IQueryable<out T> : IEnumerable<T>, IEnumerable, IQueryable

}

IQueryable<T> implements IEnumerable<T>, Its type parameter T is only used by the GetEnumerator member from IEnumerable<T>, so apparently, T remains covariant for IQueryable<T>.

Variance brings convenience to LINQ queries. Take the Concat and Select local query methods as example:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TSource> Concat<TSource>(

this IEnumerable<TSource>first, IEnumerable<TSource> second);

public static IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector);

}

In the following example, Concat is called with IEnumerable<Base> instance, so another IEnumerable<Base> instance is required. With covariance, IEnumerable<Derived> can substitute IEnumerable<Base>, so IEnumerable<Derived> argument can be directly passed to IEnumerable<base> parameter:

internal static void Concat(

IEnumerable<Base> enumerableOfBase, IEnumerable<Derived> enumerableOfDerived)

// Covariance of Concat input: IEnumerable<Derived> <: IEnumerable<Base>.

// Concat: (IEnumerable<Base>, IEnumerable<Base>) -> IEnumerable<Base>.

enumerableOfBase = enumerableOfBase.Concat(enumerableOfDerived);

}

In the following example, Select is called with IEnumerable<Derived> instance, and the output is stored as IEnumerable<Base> instance. According to the signature of Select, the selector function is required to be of Derived -> Base type. With covariance, selector function of Base -> Base, Derived -> Derived, and Base -> Derived types work as well:

internal static void Select(IEnumerable<Derived> enumerableOfDerived)

IEnumerable<Base> enumerableOfBase;

// Default with no variance.

// Select: (IEnumerable<Derived>, Derived -> Base) -> IEnumerable<Base>.

enumerableOfBase = enumerableOfDerived.Select(DerivedToBase);

// Covariance of Select input: IEnumerable<Derived> <: IEnumerable<Base>.

// Select: (IEnumerable<Base>, Base -> Base) -> IEnumerable<Base>.

enumerableOfBase = enumerableOfDerived.Select(BaseToBase);

// Covariance of Select output: IEnumerable<Derived> <: IEnumerable<Base>.

// Select: (IEnumerable<Derived>, Derived -> Derived) -> IEnumerable<Derived>.

enumerableOfBase = enumerableOfDerived.Select(DerivedToDerived);

// Covariance of Select input and output: IEnumerable<Derived> <: IEnumerable<Base>.

// Select: (IEnumerable<Base>, Base -> Derived) -> IEnumerable<Derived>.

enumerableOfBase = enumerableOfDerived.Select(BaseToDerived);

}

Summary

This chapter discusses C#’s covariance and contravariance feature in the context of non-generic delegate types, generic delegate types, generic interfaces, generic higher-order function types, as well as arrays. Variances also brings convenience to LINQ query, since LINQ follows the pattern of fluent interface, which has covariant type parameter.

C# Functional Programming In-Depth (10) Query Expression

Mon, 10 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

C# 3.0 introduces query expression, a SQL-like query syntactic sugar for query methods composition.

Syntax and compilation

The following is the syntax of query expression:

from [Type] identifier in source
[from [Type] identifier in source]
[join [Type] identifier in source on expression equals expression [into identifier]]
[let identifier = expression]
[where predicate]
[orderby ordering [ascending | descending][, ordering [ascending | descending], …]]
select expression | group expression by key [into identifier]
[continuation]

It introduces new language keywords to C#, which are called query keywords:

from
join, on, equals
let
where
orderby, ascending, descending
select
group, by
into

Query expression is compiled to query method calls at compile time:

<table border="0" cellpadding="2" cellspacing="0" width="672"><tbody><tr><td valign="top" width="297">Query expression</td><td valign="top" width="373">Query method</td></tr><tr><td valign="top" width="297">single from clause with select clause</td><td valign="top" width="373">Select</td></tr><tr><td valign="top" width="297">multiple from clauses with select clause</td><td valign="top" width="373">SelectMany</td></tr><tr><td valign="top" width="297">Type in from/join clauses</td><td valign="top" width="373">Cast</td></tr><tr><td valign="top" width="297">join clause without into</td><td valign="top" width="373">Join</td></tr><tr><td valign="top" width="297">join clause with into</td><td valign="top" width="373">GroupJoin</td></tr><tr><td valign="top" width="297">let clause</td><td valign="top" width="373">Select</td></tr><tr><td valign="top" width="297">where clauses</td><td valign="top" width="373">Where</td></tr><tr><td valign="top" width="297">orderby clause with or without ascending</td><td valign="top" width="373">OrderBy, ThenBy</td></tr><tr><td valign="top" width="297">orderby clause with descending</td><td valign="top" width="373">OrderByDescending, ThenByDescending</td></tr><tr><td valign="top" width="297">group clause</td><td valign="top" width="373">GroupBy</td></tr><tr><td valign="top" width="297">into with continuation</td><td valign="top" width="373">Nested query</td></tr></tbody></table>

It is already demonstrated how query expression syntax works for LINQ. Actually, this syntax is not specific for LINQ query or IEnumerable<T>/ParallelQuery<T>/IQueryable<T> types, but a general C# syntactic sugar. Take select clause (compiled to Select method call) as example, it can work for any type, as long as the compiler can find a Select instance method or extension method for that type. Take int as example, it does not have a Select instance method, so the following extension method can be defined to accept a selector function:

internal static partial class Int32Extensions
{
    internal static TResult Select<TResult>(this int int32, Func<int, TResult> selector) => 
        selector(int32);
}

Now select clause of query expression syntax can be applied to int:

internal static partial class QueryExpression
{
    internal static void SelectInt32()
    {
        int mapped1 = from zero in default(int) // 0
                      select zero; // 0
        double mapped2 = from three in 1 + 2 // 3
                         select Math.Sqrt(three + 1); // 2
    }
}

And they are compiled to above Select extension method call:

internal static void CompiledSelectInt32()
{
    int mapped1 = Int32Extensions.Select(default, zero => zero); // 0
    double mapped2 = Int32Extensions.Select(1 + 2, three => Math.Sqrt(three + 1)); // 2
}

More generally, Select method can be defined for any type:

internal static partial class ObjectExtensions
{
    internal static TResult Select<TSource, TResult>(this TSource value, Func<TSource, TResult> selector) => 
        selector(value);
}

Now select clause and Select method can be applied to any type:

internal static void SelectGuid()
{
    string mapped = from newGuid in Guid.NewGuid()
                    select newGuid.ToString();
}

internal static void CompiledSelectGuid()
{
    string mapped = ObjectExtensions.Select(Guid.NewGuid(), newGuid => newGuid.ToString());
}

Some tools, like Resharper, a powerful extension for Visual Studio, can help converting query expressions to query methods at design time:

Query expression pattern

To enable all the query keywords for a certain type, a set of query methods are required to be provided. The following interfaces demonstrate the signatures of the required methods for a locally queryable type:

public interface ILocal
{
    ILocal<T> Cast<T>();
}

public interface ILocal<T> : ILocal
{
    ILocal<T> Where(Func<T, bool> predicate);

    ILocal<TResult> Select<TResult>(Func<T, TResult> selector);

    ILocal<TResult> SelectMany<TSelector, TResult>(
        Func<T, ILocal<TSelector>> selector,
        Func<T, TSelector, TResult> resultSelector);

    ILocal<TResult> Join<TInner, TKey, TResult>(
        ILocal<TInner> inner,
        Func<T, TKey> outerKeySelector,
        Func<TInner, TKey> innerKeySelector,
        Func<T, TInner, TResult> resultSelector);

    ILocal<TResult> GroupJoin<TInner, TKey, TResult>(
        ILocal<TInner> inner,
        Func<T, TKey> outerKeySelector,
        Func<TInner, TKey> innerKeySelector,
        Func<T, ILocal<TInner>, TResult> resultSelector);

    IOrderedLocal<T> OrderBy<TKey>(Func<T, TKey> keySelector);

    IOrderedLocal<T> OrderByDescending<TKey>(Func<T, TKey> keySelector);

    ILocal<ILocalGroup<TKey, T>> GroupBy<TKey>(Func<T, TKey> keySelector);

    ILocal<ILocalGroup<TKey, TElement>> GroupBy<TKey, TElement>(
        Func<T, TKey> keySelector, Func<T, TElement> elementSelector);
}

public interface IOrderedLocal<T> : ILocal<T>
{
    IOrderedLocal<T> ThenBy<TKey>(Func<T, TKey> keySelector);

    IOrderedLocal<T> ThenByDescending<TKey>(Func<T, TKey> keySelector);
}

public interface ILocalGroup<TKey, T> : ILocal<T>
{
    TKey Key { get; }
}

All above methods return ILocalSource<T>, so these methods or query expression clauses can be easily composed. The above query methods are represented as instance methods. As fore mentioned, extension methods work too. This is called the query expression pattern. Similarly, the following interfaces demonstrate the signatures of the required query methods for a remotely queryable type, which replaces all function parameters with expression tree parameters:

public interface IRemote
{
    IRemote<T> Cast<T>();
}

public interface IRemote<T> : IRemote
{
    IRemote<T> Where(Expression<Func<T, bool>> predicate);

    IRemote<TResult> Select<TResult>(Expression<Func<T, TResult>> selector);

    IRemote<TResult> SelectMany<TSelector, TResult>(
        Expression<Func<T, IRemote<TSelector>>> selector,
        Expression<Func<T, TSelector, TResult>> resultSelector);

    IRemote<TResult> Join<TInner, TKey, TResult>(
        IRemote<TInner> inner,
        Expression<Func<T, TKey>> outerKeySelector,
        Expression<Func<TInner, TKey>> innerKeySelector,
        Expression<Func<T, TInner, TResult>> resultSelector);

    IRemote<TResult> GroupJoin<TInner, TKey, TResult>(
        IRemote<TInner> inner,
        Expression<Func<T, TKey>> outerKeySelector,
        Expression<Func<TInner, TKey>> innerKeySelector,
        Expression<Func<T, IRemote<TInner>, TResult>> resultSelector);

    IOrderedRemote<T> OrderBy<TKey>(Expression<Func<T, TKey>> keySelector);

    IOrderedRemote<T> OrderByDescending<TKey>(Expression<Func<T, TKey>> keySelector);

    IRemote<IRemoteGroup<TKey, T>> GroupBy<TKey>(Expression<Func<T, TKey>> keySelector);

    IRemote<IRemoteGroup<TKey, TElement>> GroupBy<TKey, TElement>(
        Expression<Func<T, TKey>> keySelector, Expression<Func<T, TElement>> elementSelector);
}

public interface IOrderedRemote<T> : IRemote<T>
{
    IOrderedRemote<T> ThenBy<TKey>(Expression<Func<T, TKey>> keySelector);

    IOrderedRemote<T> ThenByDescending<TKey>(Expression<Func<T, TKey>> keySelector);
}

public interface IRemoteGroup<TKey, T> : IRemote<T>
{
    TKey Key { get; }
}

The following example demonstrates how the query expression syntax is enabled for ILocal<T> and IRemote<T>:

internal static void LocalQuery(ILocal<Uri> uris)
{
    ILocal<string> query =
        from uri in uris
        where uri.IsAbsoluteUri // ILocal.Where and anonymous method.
        group uri by uri.Host into hostUris // ILocal.GroupBy and anonymous method.
        orderby hostUris.Key // ILocal.OrderBy and anonymous method.
        select hostUris.ToString(); // ILocal.Select and anonymous method.
}

internal static void RemoteQuery(IRemote<Uri> uris)
{
    IRemote<string> query =
        from uri in uris
        where uri.IsAbsoluteUri // IRemote.Where and expression tree.
        group uri by uri.Host into hostUris // IRemote.GroupBy and expression tree.
        orderby hostUris.Key // IRemote.OrderBy and expression tree.
        select hostUris.ToString(); // IRemote.Select and expression tree.
}

Their syntax looks identical but they are compiled to different query method calls:

internal static void CompiledLocalQuery(ILocal<Uri> uris)
{
    ILocal<string> query = uris
        .Where(uri => uri.IsAbsoluteUri) // ILocal.Where and anonymous method.
        .GroupBy(uri => uri.Host) // ILocal.GroupBy and anonymous method.
        .OrderBy(hostUris => hostUris.Key) // ILocal.OrderBy and anonymous method.
        .Select(hostUris => hostUris.ToString()); // ILocal.Select and anonymous method.
}

internal static void CompiledRemoteQuery(IRemote<Uri> uris)
{
    IRemote<string> query = uris
        .Where(uri => uri.IsAbsoluteUri) // IRemote.Where and expression tree.
        .GroupBy(uri => uri.Host) // IRemote.GroupBy and expression tree.
        .OrderBy(hostUris => hostUris.Key) // IRemote.OrderBy and expression tree.
        .Select(hostUris => hostUris.ToString()); // IRemote.Select and expression tree.
}

.NET provides 3 sets of built-in query methods:

IEnumerable<T> represents local sequential data source and query, its query expression pattern is implemented by extension methods provided by System.Linq.Enumerable
ParallelQuery<T> represents local parallel data source and query, its query expression pattern is implemented by extension methods provided by System.Linq.ParallelEnumerable
IQueryable<T> represents remote data source and query, its query expression pattern is implemented by extension methods provided by System.Linq.Queryable

So query expression works for these 3 kinds of LINQ. The details of query expression usage and compilation is covered by the LINQ to Objects chapter.

Query expression vs. query method

Query expression is compiled to query method calls, either syntax can be used to build a LINQ query. However, query expression does not cover all query methods and their overloads. For example, Skip and Take query are not supported by query expression syntax:

namespace System.Linq
{
    public static class Enumerable
    {
        public static IEnumerable<TSource> Skip<TSource>(this IEnumerable<TSource> source, int count);

        public static IEnumerable<TSource> Take<TSource>(this IEnumerable<TSource> source, int count);
    }
}

The following query implement filtering and mapping queries with query expression, but Skip and Take have to be called as query methods, so it is in a hybrid syntax:

public static void QueryExpressionAndMethod(IEnumerable<Product> products)
{
    IEnumerable<string> query =
        (from product in products
         where product.ListPrice > 0
         select product.Name)
        .Skip(20)
        .Take(10);
}

Another example is, Where query method for IEnumerable<T> has 2 overloads:

namespace System.Linq
{
    public static class Enumerable
    {
        public static IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate);

        public static IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, int, bool> predicate);
    }
}

The first Where overload is supported by query expression where clause, the second overload is not.

All query expression syntax and all query methods will be discussed in detail in later chapters. Query expression is also a tool to build general functional workflow, which will also be discussed in the Category Theory chapter.

C# Functional Programming In-Depth (9) Function Composition and Chaining

Sun, 09 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

As demonstrated in the introduction chapter, in object-oriented programming, program is modelled as objects, and object composition is very common, which combines simple objects to more complex objects. In functional programming, program is modelled as functions, function composition is emphasized, which combines simple functions to build more complex functions. LINQ query is composition of quelop.

Forward composition and backward composition

In mathematics, function composition means to pass a first function f’s output to a second function g as input, which produces a new function h. The result function h is called composite function, it maps its argument x to g(f(x)), and is denoted g ∘ f. This notation can be read g circle f, g composed with f, or more intuitively, g after f. Since C# is strongly typed, if a first function’s output type is the same as the second function’s input type, then the first function’s output can be used to call the second function as input. So, a first function of type T -> TResult1 and a second function of type TResult1 -> TResult2 can be composed to a new function of type T -> TResult2:

internal static void OutputAsInput<T, TResult1, TResult2>(

Func<TResult1, TResult2> second, // TResult1 -> TResult2

Func<T, TResult1> first) // T ->TResult1

Func<T, TResult2> composition = value => second(first(value)); // T -> TResult2

}

The following example is a sequence of function calls, where a previous function’s output is passed to a next function as input:

internal static void OutputAsInput()

string @string = "-2";

int int32 = int.Parse(@string); // string -> int

int absolute = Math.Abs(int32); // int -> int

double @double = Convert.ToDouble(absolute); // int -> double

double squareRoot = Math.Sqrt(@double); // double -> double

}

The above int.Parse, Math.Abs Convert.ToDouble, and Math.Sqrt functions can be composed to a new function:

// string -> double

internal static double Composition(string value) =>

Math.Sqrt(Convert.ToDouble(Math.Abs(int.Parse(value))));

C# does not have built-in composition operators to combine functions, but the composition operation can be implemented as extension methods for Func generic delegate type:

// Input: TResult1 -> TResult2, T -> TResult1.

// Output: T -> TResult2

public static Func<T, TResult2> After<T, TResult1, TResult2>(

this Func<TResult1, TResult2> second, Func<T, TResult1> first) =>

value => second(first(value));

// Input: T -> TResult1, TResult1 -> TResult2.

// Output: T -> TResult2

public static Func<T, TResult2> Then<T, TResult1, TResult2>(

this Func<T, TResult1> first, Func<TResult1, TResult2> second) =>

value => second(first(value));

And their usage is straightforward:

internal static void Composition<T, TResult1, TResult2>(

Func<TResult1, TResult2> second, // TResult1 -> TResult2

Func<T, TResult1> first) // T -> TResult1

Func<T, TResult2> composition; // T -> TResult2

composition = second.After(first);

// Equivalent to:

composition = first.Then(second);

}

The above 2 extension methods work the same, they just provide different syntaxes. The After extension method implements the above ∘ operator. It composes the first function and the second function as second.After(first), where the right operand is called earlier, and the left operand is called later. Since reading code is from left to right, The Then extension method is implemented for a more readable syntax first.Then(second). Now the above int.Parse, Math.Abs Convert.ToDouble, and Math.Sqrt functions can be composed with After or Then::

internal static void BackwardCompositionForwardComposition()

Func<string, int> parse = int.Parse; // string -> int

Func<int, int> abs = Math.Abs; // int -> int

Func<int, double> convert = Convert.ToDouble; // int -> double

Func<double, double> sqrt = Math.Sqrt; // double -> double

// string -> double

Func<string, double> backwardComposition = sqrt.After(convert).After(abs).After(parse);

backwardComposition("-2").WriteLine(); // 1.4142135623731

// string -> double

Func<string, double> forwardComposition = parse.Then(abs).Then(convert).Then(sqrt);

forwardComposition("-2").WriteLine(); // 1.4142135623731

}

As demonstrated above, After composes functions from right to end, which is called backward composition. Then composes function from left to right, which is called forward composition. Again, backward composition and forward composition are just different syntaxes, they produce new functions doing the same work.

If functions’ output type and input type do not match, they cannot be composed directly and their design need to be adjusted. For example, .NET Standard’s list type is designed in object-oriented paradigm. The following are a few examples of its function members:

namespace System.Collections.Generic

public class List<T> : ICollection<T>, IEnumerable<T>, IEnumerable, IList<T>, IReadOnlyCollection<T>, IReadOnlyList<T>, ICollection, IList

public void Add(T item); // (List<T>, T) -> void

public void Clear(); // List<T> -> void

public List<T> FindAll(Predicate<T> match); // (List<T>, Predicate<T>) -> List<T>

public void ForEach(Action<T> action); // (List<T>, Action<T>) ->void

public void RemoveAt(int index); // (List<T>, index) -> void

public void Reverse(); // List<T> -> void

// Other members.

}

Notice FindAll accept a match function of type Predicate<T>, which is T -> bool. When FIndAll is called, it calls match function with each value in the list. If match outputs true, the value is added to the result list. Predicate<T> is equivalent to Func<T, bool>. FindAll does not use Func<T, bool> because List<T> type is released in .NET Framework 2.0, and the unified Func generic delegate types are introduced in .NET Framework 3.5. The above list functions accept different numbers of parameters. Some of them output a list and some output void. The following example operates a list in place. Apparently, these functions cannot be composed directly:

internal static void ListOperations()

List<int> list = new List<int>() { -2, -1, 0, 1, 2 };

list.RemoveAt(0); // -> void

list = list.FindAll(int32 => int32 > 0); // -> List<T>

list.Reverse(); // -> void

list.ForEach(int32 => int32.WriteLine()); // -> void

list.Clear(); // -> void

list.Add(1); // -> void

}

As discussed in the named function chapter, a type’s instance function member can be viewed as a static method with an additional first parameter of that type, which represents this reference to the current instance. For example, the type of Add looks like T -> void, and the type of Clear looks like () -> void, but since they are instance members, their types are actually (List<T>, T) -> void and List<T> -> void. To make these function composable. One refactor option is: If a function does not output list, have it output the result list; if a function has more parameter besides the list, swap the parameters so that the list parameter becomes the last parameter, and finally curry the function. The following are the transformed functions:

// // T -> List<T> -> List<T>

internal static Func<List<T>, List<T>> Add<T>(T value) =>

list => { list.Add(value); return list; };

// List<T> -> List<T>

internal static List<T> Clear<T>(List<T> list) { list.Clear(); return list; }

// Predicate<T> -> List<T> ->List<T>

internal static Func<List<T>, List<T>> FindAll<T>(Predicate<T> match) =>

list => list.FindAll(match);

// Action<T> -> List<T> ->List<T>

internal static Func<List<T>, List<T>> ForEach<T>(Action<T> action) =>

list => { list.ForEach(action); return list; };

// int -> List<T> -> List<T>

internal static Func<List<T>, List<T>> RemoveAt<T>(int index) =>

list => { list.RemoveAt(index); return list; };

// List<T> -> List<T>

internal static List<T> Reverse<T>(List<T> list) { list.Reverse(); return list; }

For example, Add is originally of type (List<T>, T) -> void. First, have Add output the manipulated list itself, which is super easy, and Add becomes (List<T>, T) ->List<T>; then swap the 2 parameters, Add becomes (T, List<T>) ->List<T>. Finally, curry Add to transformed it to T -> List<T> -> List<T>. Applying the refactor to all functions, their function types become either transformed to List<T> -> List<T>, or curried function type SomeType -> List<T> -> List<T>. Once a curried function is “partially applied” with a single argument, the result is also a function of type List<T> -> List<T>. Since all function types finally become List<T> -> List<T>, they can be composed:

internal static void TransformationForComposition()

// List<int> -> List<int>

Func<List<int>, List<int>> removeAtWithIndex = RemoveAt<int>(0);

Func<List<int>, List<int>> findAllWithPredicate = FindAll<int>(int32 => int32 > 0);

Func<List<int>, List<int>> reverse = Reverse;

Func<List<int>, List<int>> forEachWithAction = ForEach<int>(int32 => int32.WriteLine());

Func<List<int>, List<int>> clear = Clear;

Func<List<int>, List<int>> addWithValue = Add(1);

Func<List<int>, List<int>> backwardComposition =

addWithValue

.After(clear)

.After(forEachWithAction)

.After(reverse)

.After(findAllWithPredicate)

.After(removeAtWithIndex);

Func<List<int>, List<int>> forwardComposition =

removeAtWithIndex

.Then(findAllWithPredicate)

.Then(reverse)

.Then(forEachWithAction)

.Then(clear)

.Then(addWithValue);

}

So, if these functions have unified list output, and have the input list as the last parameter, then these functions can be composed with the help of partial application. This is the pattern of how to compose list operations in some functional languages:

internal static void ForwardCompositionWithPartialApplication()

Func<List<int>, List<int>> forwardComposition =

RemoveAt<int>(0)

.Then(FindAll<int>(int32 => int32 > 0))

.Then(Reverse)

.Then(ForEach<int>(int32 => int32.WriteLine()))

.Then(Clear)

.Then(Add(1));

List<int> list = new List<int>() { -2, -1, 0, 1, 2 };

List<int> result = forwardComposition(list);

}

Forward piping

Another option to compose these functions is to use forward pipe operator, which simply forward argument to function call. Again, C# does not have built-in forward pipe operator. It can be implemented as an extension method:

// Input, T, T -> TResult.

// Output TResult.

public static TResult Forward<T, TResult>(this T value, Func<T, TResult> function) =>

Its usage is also straightforward:

internal static void OutputAsInput<T, TResult1, TResult2>(

Func<TResult1, TResult2> second, // TResult1 -> TResult2

Func<T, TResult1> first, // T -> TResult1

T value)

TResult2 result = value.Forward(first).Forward(second);

}

Here the argument is forwarded to call the first function, then forward first function’s output to call the second function as input. So, the piping can go on with the third function, the fourth function, etc. This is called forward piping. For example:

internal static void ForwardPiping()

double result = "-2"

.Forward(int.Parse) // string -> int

.Forward(Math.Abs) // int -> int

.Forward(Convert.ToDouble) // int -> double

.Forward(Math.Sqrt); // double -> double

}

To pipe functions with more parameters, just partially applied the function to result a new function with single parameter, then a single argument can be forwarded to the single parameter function. The syntax also looks similar to forward composition:

internal static void ForwardPipingWithPartialApplication()

List<int> result = new List<int>() { -2, -1, 0, 1, 2 }

.Forward(RemoveAt<int>(1))

.Forward(FindAll<int>(int32 => int32 > 0))

.Forward(Reverse)

.Forward(ForEach<int>(int32 => int32.WriteLine()))

.Forward(Clear)

.Forward(Add(1));

}

The above syntax looks similar to forward composition. Forward composition and forward piping both compose functions from left to right. Their difference is that forward composition starts from the first function to be composed, it does not call any composed function, and outputs a new composite function; while forward piping starts from the argument, it directly calls the composed functions, and outputs the result.

Method chaining and fluent interface

In object-oriented programming, if a type has instance methods output that type, then these instance methods can be easily composed by just chaining the calls, for example:

internal static void InstanceMethodComposition(string @string)

string result = @string.Trim().Substring(1).Remove(10).ToUpperInvariant();

}

The above function members, Trim, Substring, Replace, ToUpperInvariant, are all instance methods of string, and they all output string. so that they can be chained. As fore mentioned, instance function member instance.Method(arg1, arg2, …) can be viewed as static function member Method(instance, arg1, arg2, …) at compile time. At runtime, unlike instance field member is allocated for each instance, instance function member is allocated only once in total just like static function member and static field member. In another word, in C#’s instance method calls syntax instance.Method(arg1, arg2, …), the . operator works like the forward pipe operator. The left side of . is a single argument for Method, the right side is the partial application of Method with other arguments. In instance method call, . just forwards the instance argument to Method function partially applied with other arguments, and finally output the result. So instance method chaining can be virtually viewed a simplified syntax of forward piping.

To compose the above list operations with method chaining. instance methods are required, and they must output list. Regarding list already have instance methods, but most of them output void, there are some options to implement method chaining. One option is to define a wrapper type to provide these instance methods for chaining:

internal class FluentList<T>

internal FluentList(List<T> list) => this.List = list;

internal List<T> List { get; }

internal FluentList<T> Add(T value) { this.List.Add(value); return this; }

internal FluentList<T> Clear() { this.List.Clear(); return this; }

internal FluentList<T> FindAll(Predicate<T> predicate) => new FluentList<T>(this.List.FindAll(predicate));

internal FluentList<T> ForEach(Action<T> action) { this.List.ForEach(action); return this; }

internal FluentList<T> RemoveAt(int index) { this.List.RemoveAt(index); return this; }

internal FluentList<T> Reverse() { this.List.Reverse(); return this; }

internal static void InstanceMethodComposition()

List<int> list = new List<int>() { -2, -1, 0, 1, 2 };

FluentList<int> resultWrapper = new FluentList<int>(list)

.RemoveAt(0)

.FindAll(int32 => int32 > 0)

.Reverse()

.ForEach(int32 => int32.WriteLine())

.Clear()

.Add(1);

List<int> result = resultWrapper.List;

}

Since C# supports extension method, which virtually adds instance method to existing type, another option is to provide list extension methods for chaining:

internal static List<T> ExtensionAdd<T>(this List<T> list, T item)

{ list.Add(item); return list; }

internal static List<T> ExtensionClear<T>(this List<T> list)

{ list.Clear(); return list; }

internal static List<T> ExtensionFindAll<T>(this List<T> list, Predicate<T> predicate) =>

list.FindAll(predicate);

internal static List<T> ExtensionForEach<T>(this List<T> list, Action<T> action)

{ list.ForEach(action); return list; }

internal static List<T> ExtensionRemoveAt<T>(this List<T> list, int index)

{ list.RemoveAt(index); return list; }

internal static List<T> ExtensionReverse<T>(this List<T> list)

{ list.Reverse(); return list; }

internal static void ExtensionMethodComposition()

List<int> result = new List<int>() { -2, -1, 0, 1, 2 }

.ExtensionRemoveAt(0)

.ExtensionFindAll(int32 => int32 > 0)

.ExtensionReverse()

.ExtensionForEach(int32 => int32.WriteLine())

.ExtensionClear()

.ExtensionAdd(1);

}

As discussed in the named function chapter, extension method is compiled to static method. Similar to instance method call, the extension method calls syntax instance.ExtensionMethod(arg1, arg2, …) can also be viewed as forwarding instance argument to static function member ExtensionMethod partially applied with other arguments. The above extension method chaining is compiled to the following static method composition:

internal static void CompiledExtensionMethodComposition()

List<int> result =

ExtensionAdd(

ExtensionClear(

ExtensionForEach(

ExtensionReverse(

ExtensionFindAll(

ExtensionRemoveAt(

new List<int>() { -2, -1, 0, 1, 2 },

),

int32 => int32> 0

),

int32 => int32.WriteLine()

),

);

}

For example, the previous forward piping of int. Parse, Math.Abs Convert.ToDouble, and Math.Sqrt functions can be simplified with extension methods:

internal static int ParseInt32(this string @string) => int.Parse(@string);

internal static int Abs(this int int32) => Math.Abs(int32);

internal static double ToDouble(this int int32) => Convert.ToDouble(int32);

internal static double Sqrt(this double @double) => Math.Sqrt(@double);

internal static void ForwardPipingWithExtensionMethod()

double result = "-2"

.ParseInt32() // .Forward(int.Parse)

.Abs() // .Forward(Math.Abs)

.ToDouble() // .Forward(Convert.ToDouble)

.Sqrt(); // .Forward(Math.Sqrt);

}

If an interface type supports method chaining, it is called fluent interface. For example, the following IAnimal interface has instance methods and extension method that output the interface type itself:

internal interface IAnimal

IAnimal Eat();

IAnimal Move();

internal static class AnimalExtensions

internal static IAnimal Sleep(this IAnimal animal) => animal;

}

All these methods can be composed by chaining, and IAnimal is a fluent interface:

internal static void FluentInterface(IAnimal animal)

IAnimal result = animal.Eat().Move().Sleep();

}

LINQ query composition

As demonstrated in the introduction chapter, LINQ query has 2 syntaxes, query method syntax and query expression syntax. They are both syntactic sugar for function composition.

LINQ query method

In the query method syntax, the LINQ query APIs are composed with the extension method chaining approach of fluent interface. In LINQ, System.Collections.Generic.IEnumerable<T> interface represents a local data source (a sequence of .NET objects) to be queried, or a local LINQ query that can be executed; System.Linq.IQueryable<T> interface represents a remote data source (e.g. data rows in a database table) to be queried, or a remote LINQ query that can be executed. The IEnumerable<T>/IQueryable<T> interfaces only have a few members, and the built-in local/remote LINQ query APIs are implemented as static function members of System.Linq.Enumerable/System.Linq.Queryable static classes. Most of these functions are extension methods for IEnumerable<T>/IQueryable<T>, and many of those extension methods output IEnumerable<T>/IQueryable<T>. For example:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> predicate);

public static IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector);

// Other members.

public static class Queryable

public static IQueryable<TSource> Where<TSource>(

this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate);

public static IQueryable<TResult> Select<TSource, TResult>(

this IQueryable<TSource> source, Expression<Func<TSource, TResult>> selector);

// Other members.

}

This kind of functions can be composed with extension method chaining. And they make IEnumerable<T>/IQueryable<T> fluent interface.

The ordering functions are slightly different. OrderBy/OrderByDescending are extension methods of IEnumerable<T>/IQueryable<T>. However, they output IOrderedEnumerable<T>/IOrderedQueryable<T>. which represent ordered data source or ordered query, and these interfaces implement IEnumerable<T>/IQueryable<T>. LINQ also provides ThenBy/ThenByDesending to perform subsequent ordering on an ordered data source or ordered query, so ThenBy/ThenByDesending are extension methods of IOrderedEnumerable<T>/IOrderedQueryable<T>:

namespace System.Linq

public interface IOrderedEnumerable<out TElement> : IEnumerable<TElement>, IEnumerable

IOrderedEnumerable<TElement> CreateOrderedEnumerable<TKey>(

Func<TElement, TKey> keySelector, IComparer<TKey> comparer, bool descending);

public static class Enumerable

public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> OrderByDescending<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> ThenBy<TSource, TKey>(

this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IOrderedEnumerable<TSource> ThenByDescending<TSource, TKey>(

this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public interface IOrderedQueryable<out T> : IEnumerable<T>, IEnumerable, IOrderedQueryable, IQueryable, IQueryable<T>

public static class Queryable

public static IOrderedQueryable<TSource> OrderBy<TSource, TKey>(

this IQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector);

public static IOrderedQueryable<TSource> OrderByDescending<TSource, TKey>(

this IQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector);

public static IOrderedQueryable<TSource> ThenBy<TSource, TKey>(

this IOrderedQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector);

public static IOrderedQueryable<TSource> ThenByDescending<TSource, TKey>(

this IOrderedQueryable<TSource> source, Expression<Func<TSource, TKey>> keySelector);

}

With this design, OrderBy/OrderByDescending can be chained after other query methods which output IEnumerable<T>/IQueryable<T>, but ThenBy/ThenByDesending can only be chained right after OrderBy/OrderByDescending which output IOrderedEnumerable<T>/IOrderedQueryable<T>. Regarding ThenBy/ThenByDesending perform subsequent ordering, this totally make sense. All the other non-ordering extension methods can be chained after OrderBy/OrderByDescending and ThenBy/ThenByDesending, since the output IOrderedEnumerable<T> is an IEnumerable<T> and IOrderedQueryable<T> is an IQueryable<T>.

Some APIs are not extension methods but static methods that output IEnumerable<T>. These static methods can be called to start query composition.

namespace System.Linq

public static class Enumerable

public static IEnumerable<TResult> Empty<TResult>();

public static IEnumerable<TResult> Repeat<TResult>(TResult element, int count);

}

There are other query APIs which do not output IEnumerable<T>/IQueryable<T>. They can be used to end the query composition. For example:

namespace System.Linq

public static class Enumerable

public static TSource First<TSource>(this IEnumerable<TSource> source);

public static int Count<TSource>(this IEnumerable<TSource> source);

public static class Queryable

public static TSource First<TSource>(this IQueryable<TSource> source);

public static int Count<TSource>(this IQueryable<TSource> source);

}

The following is an example of query composition:

internal static void LinqComposition()

int queryResultCount = Enumerable

.Repeat(0, 10) // -> IEnumerable<int>

.Select(int32 => Path.GetRandomFileName()) // -> IEnumerable<string>

.OrderByDescending(@string => @string.Length) // -> IOrderedEnumerable<string>

.ThenBy(@string => @string) // -> IOrderedEnumerable<string>

.Select(@string => $"{@string.Length}: {@string}") // -> IEnumerable<string>

.Count(); // -> int

}

Apparently, the above extension method chaining is compiled to the following function composition:

internal static void CompiledLinqComposition()

string firstQueryResult =

Enumerable.First(

Enumerable.Select(

Enumerable.ThenBy(

Enumerable.OrderByDescending(

Enumerable.Select(

Enumerable.Repeat(0, 10),

int32 => Path.GetRandomFileName()

),

@string => @string.Length

),

@string => @string

),

@string => $"{@string.Length}: {@string}"

);

}

So, in query method syntax, LINQ query is represented by fluent interface and query methods are composed with extension method chaining. This design makes LINQ easy to use, as well as extensible. The above built-in functions provided by Enumerable/Queryable are called LINQ standard query methods or standard query operators. Developers can also implement custom query APIs as extension methods of IEnumerable<T>/IQueryable<T> and compose built-in query methods and custom query methods by chaining. The details of local and remote LINQ query methods are discussed in the LINQ chapters.

LINQ query expression

LINQ query expression is a SQL/XQuery-like declarative query syntactic sugar for LINQ query composition. As an expression, it is composed with clauses. The following is the syntax of query expression:

from [Type] rangeVariable in source

[from [Type] rangeVariable in source]

[join [Type] rangeVariable in source on outerKey equals innerKey [into rangeVariable]]

[let rangeVariable = expression]

[where predicate]

[orderby orderingKey [ascending | descending][, orderingKey [ascending | descending], …]]

select projection | group projection by groupKey [into rangeVariable

continuationClauses]

A query expression must start with a from clause, and end with either a select clause or a group clause. In the middle, it can have from/join/let/where/orderby clauses. These query expression clauses introduce new language keywords to C#, which are called query keywords:

· from

· join, on, equals

· let

· where

· orderby, ascending, descending

· select

· group, by

· into

Query expression is just syntactic sugar, it is compiled to query method calls:

The local-variable-like identifiers declared in query expression are called range variables. The scope of range variable ends either at the end of query expression or at the into keyword that begins a continuation clause. For example:

internal static void RangeVariable(IEnumerable<int> source)

IEnumerable<string> query =

from variable in source // variable is int.

where variable > 0 // variable is int.

select variable.ToString() /* variable is int. */ into variable // variable is string.

orderby variable.Length // variable is string.

select variable.ToUpperInvariant(); // variable is string.

}

The from clause declares range variable of type int, which represents each value in the int source sequence. The variable can be used in the next clauses until select clause’s into keyword. The into keyword is followed by a different range variable of type string, and can be used until the end of query, since there is no further continuation.

The following example demonstrate the syntax of each kind of clauses in local query, and the query methods they are compiled to:

internal static void QueryExpressionClause(

IEnumerable<int> int32Sequence, IEnumerable<string> stringSequence)

IEnumerable<int> singleFromWithSelect;

singleFromWithSelect = from int32 in int32Sequence

select int32;

// IEnumerable<TResult> Select<TSource, TResult>(this IEnumerable<TSource> source, Func<TSource, TResult> selector);

singleFromWithSelect = int32Sequence.Select(int32 => int32);

IEnumerable<string> let;

let = from int32 in int32Sequence

let variable = int32 + 1

select variable.ToString();

// IEnumerable<TResult> Select<TSource, TResult>(this IEnumerable<TSource> source, Func<TSource, TResult> selector);

let = int32Sequence

.Select(int32 => new { int32, variable = int32 + 1 })

.Select(context => context.variable.ToString());

IEnumerable<int> multipleFromWithSelect;

multipleFromWithSelect = from int32 in int32Sequence

from @string in stringSequence

select int32 + @string.Length;

// IEnumerable<TResult> SelectMany<TSource, TCollection, TResult>(this IEnumerable<TSource> source, Func<TSource, IEnumerable<TCollection>> collectionSelector, Func<TSource, TCollection, TResult> resultSelector);

multipleFromWithSelect = int32Sequence.SelectMany(

int32 => stringSequence, (int32, @string) => int32 + @string.Length);

IEnumerable<DayOfWeek> typeInFrom;

typeInFrom = from DayOfWeek dayOfWeek in int32Sequence

select dayOfWeek;

// IEnumerable<TResult> Cast<TResult>(this IEnumerable source);

typeInFrom = int32Sequence.Cast<DayOfWeek>().Select(dayOfWeek => dayOfWeek);

IEnumerable<int> where;

where = from int32 in int32Sequence

where int32 > 0

select int32;

// IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, bool> predicate);

where = int32Sequence.Where(int32 => int32 > 0);

IOrderedEnumerable<string> orderByWithSingleKey;

orderByWithSingleKey = from @string in stringSequence

orderby @string.Length

select @string;

// IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

orderByWithSingleKey = stringSequence.OrderBy(@string => @string.Length);

IOrderedEnumerable<string> orderByWithMultipleKeys;

orderByWithMultipleKeys = from @string in stringSequence

orderby @string.Length, @string descending

select @string;

// IOrderedEnumerable<TSource> ThenByDescending<TSource, TKey>(this IOrderedEnumerable<TSource> source, Func<TSource, TKey> keySelector);

orderByWithMultipleKeys = stringSequence

.OrderBy(@string => @string.Length).ThenByDescending(@string => @string);

IEnumerable<IGrouping<int, int>> group;

group = from int32 in int32Sequence

group int32 by int32 % 10;

// IEnumerable<IGrouping<TKey, TSource>> GroupBy<TSource, TKey>(this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

group = int32Sequence.GroupBy(int32 => int32 % 10);

IEnumerable<IGrouping<int, string>> groupWithElementSelector;

groupWithElementSelector = from int32 in int32Sequence

group int32.ToString() by int32 % 10;

// IEnumerable<IGrouping<TKey, TElement>> GroupBy<TSource, TKey, TElement>(this IEnumerable<TSource> source, Func<TSource, TKey> keySelector, Func<TSource, TElement> elementSelector);

groupWithElementSelector = int32Sequence.GroupBy(int32 => int32 % 10, int32 => int32.ToString());

IEnumerable<string> join;

join = from int32 in int32Sequence

join @string in stringSequence on int32 equals @string.Length

select $"{int32} - {@string}";

// IEnumerable<TResult> Join<TOuter, TInner, TKey, TResult>(this IEnumerable<TOuter> outer, IEnumerable<TInner> inner, Func<TOuter, TKey> outerKeySelector, Func<TInner, TKey> innerKeySelector, Func<TOuter, TInner, TResult> resultSelector);

join = int32Sequence.Join(

stringSequence,

int32 => int32,

@string => @string.Length,

(int32, @string) => $"{int32} - {@string}");

IEnumerable<string> joinWithInto;

joinWithInto = from int32 in int32Sequence

join @string in stringSequence on int32 equals @string.Length into stringGroup

select $"{int32} - {string.Join(", ", stringGroup)}";

// IEnumerable<TResult> GroupJoin<TOuter, TInner, TKey, TResult>(this IEnumerable<TOuter> outer, IEnumerable<TInner> inner, Func<TOuter, TKey> outerKeySelector, Func<TInner, TKey> innerKeySelector, Func<TOuter, IEnumerable<TInner>, TResult> resultSelector);

joinWithInto = int32Sequence.GroupJoin(

stringSequence,

int32 => int32,

@string => @string.Length,

(int32, stringGroup) => $"{int32} - {string.Join(", ", stringGroup)}");

IEnumerable<IGrouping<char, string>> intoWithContinuation;

intoWithContinuation = from int32 in int32Sequence

select Math.Abs(int32) into absolute

select absolute.ToString() into @string

group @string by @string[0];

intoWithContinuation = int32Sequence

.Select(int32 => Math.Abs(int32))

.Select(absolute => absolute.ToString())

.GroupBy(@string => @string[0]);

}

The compilation of remote LINQ querie expressions works in the same way. The details of these query methods are discussed in the LINQ chapters. Query expressions composed with multiple clauses is compiled to query methods composition:

internal static void QueryExpression(

IEnumerable<int> int32Sequence, IEnumerable<string> stringSequence)

IEnumerable<IGrouping<char, ConsoleColor>> query;

query = from int32 in int32Sequence

from @string in stringSequence // SelectMany.

let length = @string.Length // Select.

where length > 1 // Where.

select int32 + length

into sum // Select.

join ConsoleColor color in int32Sequence on sum % 15 equals (int)color // Join.

orderby color // OrderBy.

group color by color.ToString()[0]; // GroupBy.

query = int32Sequence

.SelectMany(int32 => stringSequence, (int32, @string) => new { int32, @string }) // Multiple from clauses.

.Select(context => new { context, length = context.@string.Length }) // let clause.

.Where(context => context.length > 1) // where clause.

.Select(context => context.context.int32 + context.length) // select clause.

.Join(

int32Sequence.Cast<ConsoleColor>(),

sum => sum % 15,

color => (int)color,

(sum, color) => new { sum, color }) // join clause without into.

.OrderBy(context => context.color) // orderby clause.

.GroupBy(

context => context.color.ToString()[0],

context => context.color); // group by clause without element selector.

}

The both LINQ syntaxes are compiled to the following function composition, where one query API’s out put is passed to another query API as input:

internal static void CompiledQueryExpression(

IEnumerable<int> int32Sequence, IEnumerable<string> stringSequence)

IEnumerable<IGrouping<char, ConsoleColor>> query =

Enumerable.GroupBy(

Enumerable.OrderBy(

Enumerable.Join(

Enumerable.Select(

Enumerable.Where(

Enumerable.Select(

Enumerable.SelectMany(

int32Sequence, int32 => stringSequence,

(int32, @string) => new { int32, @string }),

context => new { context, length = context.@string.Length }

),

context => context.length > 1

),

context => context.context.int32 + context.length

),

Enumerable.Cast<ConsoleColor>(int32Sequence),

sum => sum % 15,

color => (int)color,

(sum, color) => new { sum, color }

),

context => context.color

),

context => context.color.ToString()[0],

context => context.color

);

}

This comparison demonstrates that query expression and query method chaining are great syntactic sugar to write declarative and functional code.

Some tools, like Resharper, an extension for Visual Studio, utilizes C# compiler to convert query expressions to query methods at design time:

Query expression pattern

LINQ query expression is also an extensible model. In the above examples, query expression for IEnumerable<T> is compiled to calls of query method for IEnumerable<T>. Just like extension method chaining can be implemented for any type, query expression can be available for any type as well. If the compiler can find a query method exists for a type, the compiler supports the corresponding query expression clause’s syntax for that type. For example, the following example implements Select query method for int and System.Guid types:

private static TResult Select<TResult>(this int source, Func<int, TResult> selector) =>

selector(source);

private static TResult Select<TResult>(this Guid source, Func<Guid, TResult> selector) =>

selector(source);

Similar to Enumerable.Select for IEnumerable<T>, the above Select methods are also extension methods for int/Guid. They accept a function parameter, and output a result. Now C# compiler supports query expression of single from clause with select clause for int/Guid. The following is the query expression syntax and the equivalent query method syntax:

internal static void Select()

int defaultInt32;

defaultInt32 = from zero in default(int)

select zero;

defaultInt32 = Select(default(int), zero => zero);

double squareRoot;

squareRoot = from three in 1 + 2

select Math.Sqrt(three + 1);

squareRoot = Select(1 + 2, three => Math.Sqrt(three + 1));

string guidString;

guidString = from guid in Guid.NewGuid()

select guid.ToString();

guidString = Select(Guid.NewGuid(), guid => guid.ToString());

}

If Where is implemented for int, then C# compiler supports where clause for int. If SelectMany is implemented for Guid, then C# compiler supports multiple from clauses for Guid:

private static int? Where(this int source, Func<int, bool> predicate) =>

predicate(source) ? (int?)source : null;

private static TResult SelectMany<TSelector, TResult>(

this Guid source, Func<Guid, TSelector> selector, Func<Guid, TSelector, TResult> resultSelector)

TSelector selectorResult = selector(source);

return resultSelector(source, selectorResult);

internal static void WhereAndSelectMany()

int? positive;

positive = from random in new Random().Next()

where random > 0

select random;

positive = new Random().Next().Where(random => random > 0);

string doubleGuidString;

doubleGuidString = from guild1 in Guid.NewGuid()

from guid2 in Guid.NewGuid()

select guild1.ToString() + guid2.ToString();

doubleGuidString = Guid.NewGuid().SelectMany(

guild1 => Guid.NewGuid(), (guild1, guid2) => guild1.ToString() + guid2.ToString());

}

If more query methods are implemented for a type, more query expression clauses are supported for that type. And, to compose the clauses, the query methods must be able to fluently chainable, which means their input type and output type must be matching a pattern. The following are the type designs and query method input/output designs needed to make all local LINQ query expression clauses available and composable. This is called the query expression pattern of C#:

public interface ILocal

ILocal<T> Cast<T>();

public interface ILocal<T> : ILocal

ILocal<TResult> Select<TResult>(Func<T, TResult> selector); // select clause.

ILocal<TResult> SelectMany<TSelector, TResult>(

Func<T, ILocal<TSelector>> selector,

Func<T, TSelector, TResult> resultSelector); // Multiple from clause.

ILocal<T> Where(Func<T, bool> predicate); // where clause.

IOrderedLocal<T> OrderBy<TKey>(Func<T, TKey> keySelector); // orderby clause.

IOrderedLocal<T> OrderByDescending<TKey>(Func<T, TKey> keySelector); // orderby clause with descending.

ILocal<ILocalGroup<TKey, T>> GroupBy<TKey>(Func<T, TKey> keySelector); // group clause without element selector.

ILocal<ILocalGroup<TKey, TElement>> GroupBy<TKey, TElement>(

Func<T, TKey> keySelector, Func<T, TElement> elementSelector); // group clause with element selector.

ILocal<TResult> Join<TInner, TKey, TResult>(

ILocal<TInner> inner,

Func<T, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<T, TInner, TResult> resultSelector); // join clause.

ILocal<TResult> GroupJoin<TInner, TKey, TResult>(

ILocal<TInner> inner,

Func<T, TKey> outerKeySelector,

Func<TInner, TKey> innerKeySelector,

Func<T, ILocal<TInner>, TResult> resultSelector); // join clause with into.

public interface IOrderedLocal<T> : ILocal<T>

IOrderedLocal<T> ThenBy<TKey>(Func<T, TKey> keySelector); // Multiple keys in orderby clause.

IOrderedLocal<T> ThenByDescending<TKey>(Func<T, TKey> keySelector); // Multiple keys with descending in orderby clause.

public interface ILocalGroup<TKey, T>

TKey Key { get; }

}

The above query methods are demonstrated as instance methods for convenience. As fore mentioned, extension methods work too. Similarly, the following type design and query method design demonstrate the remote LINQ query expression pattern, where all function parameters are replaced with corresponding expression tree parameters:

public interface IRemote

IRemote<T> Cast<T>();

public interface IRemote<T> : IRemote

IRemote<TResult> Select<TResult>(Expression<Func<T, TResult>> selector);

IRemote<TResult> SelectMany<TSelector, TResult>(

Expression<Func<T, IRemote<TSelector>>> selector,

Expression<Func<T, TSelector, TResult>> resultSelector);

IRemote<T> Where(Expression<Func<T, bool>> predicate);

IOrderedRemote<T> OrderBy<TKey>(Expression<Func<T, TKey>> keySelector);

IOrderedRemote<T> OrderByDescending<TKey>(Expression<Func<T, TKey>> keySelector);

IRemote<IRemoteGroup<TKey, T>> GroupBy<TKey>(Expression<Func<T, TKey>> keySelector);

IRemote<IRemoteGroup<TKey, TElement>> GroupBy<TKey, TElement>(

Expression<Func<T, TKey>> keySelector, Expression<Func<T, TElement>> elementSelector);

IRemote<TResult> Join<TInner, TKey, TResult>(

IRemote<TInner> inner,

Expression<Func<T, TKey>> outerKeySelector,

Expression<Func<TInner, TKey>> innerKeySelector,

Expression<Func<T, TInner, TResult>> resultSelector);

IRemote<TResult> GroupJoin<TInner, TKey, TResult>(

IRemote<TInner> inner,

Expression<Func<T, TKey>> outerKeySelector,

Expression<Func<TInner, TKey>> innerKeySelector,

Expression<Func<T, IRemote<TInner>, TResult>> resultSelector);

public interface IOrderedRemote<T> : IRemote<T>

IOrderedRemote<T> ThenBy<TKey>(Expression<Func<T, TKey>> keySelector);

IOrderedRemote<T> ThenByDescending<TKey>(Expression<Func<T, TKey>> keySelector);

public interface IRemoteGroup<TKey, T>

TKey Key { get; }

}

.NET Standard provides 3 sets of built-in query APIs, implementing the above query expression pattern:

· The IEnumerable, IEnumerable<T>, IOrderedEnumerable<T> and IGrouping<TKey, TElement> types follow the above type designs, and Enumerable’s function members follow the above query method designs and implement local sequential queries.

· The ParallelQuery, ParallelQuery<TSource>, OrderedParallelQuery<TSource> and IGrouping<TKey, TElement> types follow the above type designs, and System.Linq.ParallelEnumerable’s function members follow the above query method designs and implement local parallel queries.

· The IQueryable, IQueryable<T>, IOrderedQueryable<T> and IGrouping<TKey, TElement> types follow the above type designs, and Queryable’s function members follow the above query method designs and are used for remote queries.

The details of local sequential queries are discussed in LINQ to Objects chapters, local parallel queries are discussed in the Parallel LINQ chapters, and the remote queries are discussed in the LINQ to Entities chapters.

Forward piping with LINQ

As fore mentioned, LINQ query method chaining or query expression is essentially function composition through forward piping. General forward piping can be also implemented by LINQ. One option is to use Select. The Select query method for IEnumerable<T> forwards each value in the IEnumerable<T> sequence to the selector function. In the previous examples, The Select query method for int/Guid forwards the int/Guid value to the selector function too. Following this pattern, a generic Select query method can be implemented for all types:

private static TResult Select<TSource, TResult>(

this TSource source, Func<TSource, TResult> selector) =>

selector(source);

This version of Select becomes the same as the Forward operator, so the previous example’s Forward calls can be replaced by Select calls. For query expression, 1 Select call can be compiled from a select clause. 2 chaining Select calls can be compiled from a select clause with into and continuation of another select clauses. So the equivalent query expression version of forward piping can be by multiple select clauses with into:

internal private static TResult Select<TSource, TResult>(

this TSource source, Func<TSource, TResult> selector) =>

selector(source);

internal static void ForwardPipingWithSelect()

double squareRoot;

squareRoot = from @string in "-2"

select int.Parse(@string) into int32

select Math.Abs(int32) into absolute

select Convert.ToDouble(absolute) into @double

select Math.Sqrt(@double);

squareRoot = "-2"

.Select(int.Parse)

.Select(Math.Abs)

.Select(Convert.ToDouble)

.Select(Math.Sqrt);

}

The other option is SelectMany. Instead of multiple select clauses, SelectMany enables multiple from clauses to implement forward piping. The following are the generic SelectMany query method for all types, the forward piping query expression with multiple from clauses, and the equivalent query method call version:

private static TResult SelectMany<TSource, TSelector, TResult>(

this TSource source,

Func<TSource, TSelector> selector,

Func<TSource, TSelector, TResult> resultSelector)

TSelector selectorResult = selector(source);

return resultSelector(source, selectorResult);

internal static void ForwardPipingWithQueryExpression()

double result;

result = from @string in "-2"

from int32 in int.Parse(@string)

from absolute in Math.Abs(int32)

from @double in Convert.ToDouble(absolute)

from squareRoot in Math.Sqrt(@double)

select squareRoot;

result = "-2"

.SelectMany(

@string => int.Parse(@string),

(@string, int32) => new { @string, int32 })

.SelectMany(

context => Math.Abs(context.int32),

(context, absolute) => new { context, absolute })

.SelectMany(

context => Convert.ToDouble(context.absolute),

(context, @double) => new { context, @double })

.SelectMany(

context => Math.Sqrt(context.@double),

(context, squareRoot) => squareRoot);

}

Query expression vs. query method

Query expression is compiled to query method calls. Query expression is declarative and intuitive, but it has 2 disadvantages. First, query expression does not cover all the built-in query methods. For example, Skip and Take query are not supported by query expression syntax:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TSource> Skip<TSource>(this IEnumerable<TSource> source, int count);

public static IEnumerable<TSource> Take<TSource>(this IEnumerable<TSource> source, int count);

}

With query expression, to skip a number of values and take a number of values, a hybrid syntax has to be used:

internal static void QueryExpressionAndQueryMethod(IEnumerable<int> source)

IEnumerable<int> query =

(from int32 in source

where int32 > 0

select int32)

.Skip(20)

.Take(10);

}

Second, for the covered query methods, not all query method overloads are supported. For example, Where has 2 overloads:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, bool>predicate);

public static IEnumerable<TSource> Where<TSource>(this IEnumerable<TSource> source, Func<TSource, int, bool> predicate);

}

The first Where overload is supported by query expression, and can be compiled from where clause. The second overload accepts a predicate function with index input. It is not supported by query expression syntax, and can only used with query method call:

internal static void WhereWithIndex(IEnumerable<int> source)

IEnumerable<int> query = source.Where((int32, index) => int32> 0 && index % 2 == 0);

}

Summary

This chapter discusses functional composition. In functional programming, function can be composed with forward composition, backward composition, and forward piping. In C#, LINQ query is a composition of query method, it is a forward piping implemented by extension method chaining. LINQ also supports query expression. Query expression is composed with clauses, and compiled to query methods chaining.

C# Functional Programming In-Depth (8) Higher-order Function, Currying and First Class Function

Sat, 08 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

The delegate chapter and other previous chapters have demonstrated that in C#, function supports many operations that are available for object. This chapter discusses one more aspect, higher-order function, and how it and other functional features make function the first-class citizen in C# language.

First-order function and higher-order function

Higher-order function is a function that accept one or more function parameters as input, or output a function. In contrast, function does not input or output any function is called first-order function. C# supports higher-order function from the beginning, because C# function can have almost any data type and function type as its input and output, except:

· Static classes, like System.Convert, System.Math, etc., because they cannot be instantiated.

· Special type System.Void.

This chapter use the following simple data type and simple function type for demonstration:

internal partial class Data

internal Data(int value) => this.Value = value;

internal int Value { get; }

// () -> void.

internal delegate void Function();

A first-order function can have normal data value as input and output:

internal static void CallFirstOrderFunction()

Data FirstOrderFunction(Data value) { return value; }

Data input = default;

Data output = FirstOrderFunction(input);

}

If a function has function as input and output, it is a higher-order function:

internal static void CallNamedHigherOrderFunction()

Function NamedHigherOrderFunction(Function value) { return value; }

Function input = default;

Function output = NamedHigherOrderFunction(input);

}

Above example is a named higher-order function. Anonymous higher-order functions can also be easily defined with lambda expression:

internal static void CallAnonymousHigherOrderFunction()

Action firstOrder1 = () => nameof(firstOrder1).WriteLine();

Func<int> firstOrder2 = () => 1;

// (() -> void) -> void

// Input: function of type () -> void. Output: void.

Action<Action> higherOrder1 = action => action();

higherOrder1(firstOrder1); // firstOrder1

higherOrder1(() => nameof(higherOrder1).WriteLine()); // higherOrder1

// () -> (() -> int)

// Input: none. Output: function of type () -> int.

Func<Func<int>> higherOrder2 = () => firstOrder2;

Func<int> output2 = higherOrder2();

output2().WriteLine(); // 1

// int -> (() -> int)

// Input: value of type int. Output: function of type () -> int.

Func<int, Func<int>> higherOrder3 = int32 => new Func<int>(() => int32 + 1);

Func<int> output3 = higherOrder3(1);

output3().WriteLine(); // 2

// (() -> void, () -> int) -> (() -> bool)

// Input: function of type () -> void, function of type () -> int. Output: function of type () -> bool.

Func<Action, Func<int>, Func<bool>> higherOrder4 = (action, int32Factory) =>

action();

return () => int32Factory() > 0;

};

Func<bool> output4 = higherOrder4(firstOrder1, firstOrder2); // firstOrder1

output4().WriteLine(); // True

Func<bool> output5 = higherOrder4(() => nameof(higherOrder4).WriteLine(), () => 0); // higherOrder4

output5().WriteLine(); // False

}

These higher-order functions can be defined and called with IIFE syntax, without any function name or function variable name involved:

internal static void AnonymousHigherOrderIife()

// (() -> void) -> void

new Action<Action>(action => action())(

() => nameof(AnonymousHigherOrderIife).WriteLine()); // AnonymousHigherOrderIife

// () -> (() -> int)

Func<int> output2 = new Func<Func<int>>(() => (() => 1))();

output2().WriteLine(); // 1

// int -> (() -> int)

Func<int> output3 = new Func<int, Func<int>>(int32 => (() => int32 + 1))(1);

output3().WriteLine(); // 2

// (() -> int, () -> string) ->(() -> bool)

Func<bool> output4 = new Func<Action, Func<int>, Func<bool>>((action, int32Factory) =>

action();

return () => int32Factory() > 0;

})(arg1: () => nameof(AnonymousHigherOrderIife).WriteLine(), arg2: () => 0); // AnonymousHigherOrderIife

output4().WriteLine();

}

.NET Standard has many built in higher-order functions, like Array.FindAll:

namespace System

public abstract class Array : ICloneable, IList, ICollection, IEnumerable, IStructuralComparable, IStructuralEquatable

public static T[] FindAll<T>(T[] array, Predicate<T>match);

}

It iterates all values in the input array, and call the match function for each value. If match function returns true, the value is added to the result array:

internal static void FilterArray(Data[] array)

Data[] filtered = Array.FindAll(array, data => data != null);

}

Many LINQ query methods are higher-order functions, take the fore mentioned Where, OrderBy, Select as example:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> predicate);

public static IOrderedEnumerable<TSource> OrderBy<TSource, TKey>(

this IEnumerable<TSource> source, Func<TSource, TKey> keySelector);

public static IEnumerable<TResult> Select<TSource, TResult>(

this IEnumerable<TSource> source, Func<TSource, TResult> selector);

public static class Queryable

public static IQueryable<TSource> Where<TSource>(

this IQueryable<TSource> source, Func<TSource, bool> predicate);

public static IOrderedQueryable<TSource> OrderBy<TSource, TKey>(

this IQueryable<TSource> source, Func<TSource, TKey> keySelector);

public static IQueryable<TResult> Select<TSource, TResult>(

this IQueryable<TSource> source, Func<TSource, TResult> selector);

}

The Where, OrderBy, Select query methods for local LINQ query are higher-order functions, since they accept an IEnumerable<T> local data source and a function as input. The query methods for remote LINQ query are first-order functions, since they accept an IQueryable<T> remote data source and an expression tree data structure. These LINQ query methods will be discussed in detail in the LINQ to Objects chapters and LINQ to Entities chapters.

Convert first-order function to higher-order function

It is possible to convert a first-order to higher-order function. In the following example, function add2Args and higherOrderAdd2Args do the same work – add 2 int values and output the sum:

internal static void Add2ArgsFirstOrderToHigherOrder()

// (int, int) -> int

Func<int, int, int> add2Args = (a, b) => a + b;

int result = add2Args(1, 2); // 3

// int -> (int -> int)

// Input: value of type int. output: function of type int -> int.

Func<int, Func<int, int>> higherOrderAdd2Args = a =>

new Func<int, int>(b => a + b);

Func<int, int> add1ArgAnd1Variable = higherOrderAdd2Args(1); // Equivalent to: b => 1 + b.

int higherOrderResult = add1ArgAnd1Variable(2); // 3

}

Apparently, add2Args is first-order function of type (int, int) –> int. It accepts the first and the second int values as input, and outputs their sum. In contrast, higherOrderAdd2Args is higher-order function of type int –> (int –> int). It accepts only the first int value as input, and outputs a function of type int –> int. This new function accepts the second int value as input, adds with the first int value as free variable captured by closure: b => 1 + b, and outputs their sum. When using add2Args, there is 1 call, where both the first and second int values must be provided, and the result is directly returned. When using higherOrderAdd2Args, there are 2 calls and an intermediate function involved, where only 1 int value is required for each call.

Similarly, a first-order function with 3 parameters can be converted to higher-order function with the same pattern:

internal static void Add3ArgsFirstOrderToHigherOrder()

// (int, int, int) -> int

Func<int, int, int, int> add3Args = (a, b, c) => a + b + c;

int result = add3Args(1, 2, 3); // 6

// int ->(int -> (int -> int))

// Input: value of type int. output: function of type int -> (int -> int), the same as above higherOrderSumOfTwoIntegers.

Func<int, Func<int, Func<int, int>>> higherOrderAdd3Args = a =>

new Func<int, Func<int, int>>(b =>

new Func<int, int>(c => a + b + c));

Func<int, Func<int, int>> higherOrderAdd2ArgsAnd1Variable = higherOrderAdd3Args(1); // Equivalent to: b => (c => 1 + b + c).

Func<int, int> add1ArgAnd2Variables = higherOrderAdd2ArgsAnd1Variable(2); // Equivalent to: c => 1 + 2 + c.

int higherOrderResult = add1ArgAnd2Variables(3); // 6

}

Again, when using first-order function, there is only 1 call, where all 3 arguments must be provided. This time, when using the higher-order function, there are 3 calls, and 2 intermediate functions, where only 1 argument is required for each call.

In the above examples, since the higher-order functions’ type is provided, C# can infer the returned intermediate functions’ type information. So, the lambda expressions can be simplified:

internal static void TypeInference()

// int -> (int -> int)

Func<int, Func<int, int>> higherOrderAdd2Args = a => (b => a + b);

// int -> (int -> (int -> int))

Func<int, Func<int, Func<int, int>>> higherOrderAdd3Args = a => (b => (c => a + b + c));

}

Lambda operator associativity

In C#, the lambda operator can be viewed as right associative. For the above higher-order functions, the above parenthesis on the right side of lambda operator can be omitted. Now compare the first-order functions and the converted higher-order functions:

internal static void LambdaOperatorAssociativity()

// (int, int) -> int

Func<int, int, int> add2Args = (a, b) => a + b;

// int ->int -> int

Func<int, Func<int, int>> higherOrderAdd2Args = a => b => a + b;

// (int, int, int) -> int

Func<int, int, int, int> add3Args = (a, b, c) => a + b + c;

// int ->int -> int -> int

Func<int, Func<int, Func<int, int>>> higherOrderAdd3Args = a => b => c => a + b + c;

}

In C#, for a function with 2 or more arguments, it is really easy to convert to equivalent higher-order function. Following this syntax, in this book, the function type notation’s -> operator is also right associative, parenthesis on the right side of -> can also be omitted. So int -> (int -> int) is identical to int -> int -> int. Since ->is not left associative, they are different from (int -> int) -> int, which accepts a function of int -> int as input, and outputs int. Similarly, int -> (int -> (int -> int)) is identical to int -> int -> int -> int.

Curry function

As demonstrated above, with higher-order function and closure, a function with 2 or more parameters can be easily converted to a sequence of nested single parameter functions, and single function call is also converted to a chain of function calls. This transformation is called currying. The term "currying" is introduced by Christopher Strachey in 1967, which is the last name of mathematician and logician Haskell Curry.

Generally, a function with N parameters can be curried to a sequence of N nested functions with single parameter:

internal static void CurryFunc<T1, T2, T3, TN, TResult>()

// (T1, T2, T3, T4, ... TN) -> TResult

Func<T1, T2, T3, /* T4, ... */ TN, TResult> function =

(value1, value2, value3, /* value4, ... */ valueN) => default;

// T1 -> T2 -> T3 -> ... TN ->TResult

Func<T1, Func<T2, Func<T3, /* Func<T4, ... */ Func<TN, TResult> /* ...> */>>> curriedFunction =

value1 => value2 => value3 => /* value4 => ... */ valueN => default;

}

The above transformation can be implemented as the following Curry extension methods for Func generic delegate types:

// Transform (T1, T2) ->TResult

// to T1 -> T2 -> TResult.

public static Func<T1, Func<T2, TResult>> Curry<T1, T2, TResult>(

this Func<T1, T2, TResult> function) =>

value1 => value2 => function(value1, value2);

// Transform (T1, T2, T3) -> TResult

// to T1 -> T2 -> T3 -> TResult.

public static Func<T1, Func<T2, Func<T3, TResult>>> Curry<T1, T2, T3, TResult>(

this Func<T1, T2, T3, TResult> function) =>

value1 => value2 => value3 => function(value1, value2, value3);

// Transform (T1, T2, T3, T4) => TResult

// to T1 -> T2 -> T3 -> T4 -> TResult.

public static Func<T1, Func<T2, Func<T3, Func<T4, TResult>>>> Curry<T1, T2, T3, T4, TResult>(

this Func<T1, T2, T3, T4, TResult> function) =>

value1 => value2 => value3 => value4 => function(value1, value2, value3, value4);

// ...

Now function can be curried by just calling its Curry extension method:

internal static void CurryFunction()

// (int, int) -> int

Func<int, int, int> add2Args = (a, b) => a + b;

int add2ArgsResult = add2Args(1, 2);

// int -> int -> int

Func<int, Func<int, int>> curriedAdd2Args = add2Args.Curry();

int curriedAdd2ArgsResult = curriedAdd2Args(1)(2);

// (int, int, int) -> int

Func<int, int, int, int> add3Args = (a, b, c) => a + b + c;

int add3ArgsResult = add2Args(1, 2);

// int -> int -> int -> int

Func<int, Func<int, Func<int, int>>> curriedAdd3Args = add3Args.Curry();

int curriedAdd3ArgsResult = curriedAdd3Args(1)(2)(3);

}

Function without output can be curried in the same way:

internal static void CurryAction<T1, T2, T3, TN>()

// (T1, T2, T3, ... TN) -> void

Action<T1, T2, T3, /* T4, ... */ TN> function =

(value1, value2, value3, /* value4, ... */ valueN) => { };

// T1 -> T2 -> T3 -> ... TN ->void

Func<T1, Func<T2, Func<T3, /* Func<T4, ... */ Action<TN>/* ...> */>>> curriedFunction =

value1 => value2 => value3 => /* value4 => ... */ valueN => { };

}

Similarly, the above transformation can be implemented as the following Curry extension methods for Action generic delegate types:

// Transform (T1, T2) ->void

// to T1 => T2 -> void.

public static Func<T1, Action<T2>> Curry<T1, T2>(

this Action<T1, T2> function) =>

value1 => value2 => function(value1, value2);

// Transform (T1, T2, T3) -> void

// to T1 -> T2 -> T3 -> void.

public static Func<T1, Func<T2, Action<T3>>> Curry<T1, T2, T3>(

this Action<T1, T2, T3> function) =>

value1 => value2 => value3 => function(value1, value2, value3);

// Transform (T1, T2, T3, T4) -> void

// to T1 -> T2 -> T3 -> T4 -> void.

public static Func<T1, Func<T2, Func<T3, Action<T4>>>> Curry<T1, T2, T3, T4>(

this Action<T1, T2, T3, T4> function) =>

value1 => value2 => value3 => value4 => function(value1, value2, value3, value4);

// ...

Now function without output can also be curried by just calling its Curry extension method:

Internal static void CurryAction()

// (int, int) -> void

Action<int, int> add2Args = (a, b) => (a + b).WriteLine();

add2Args(1, 2);

// int -> int -> void

Func<int, Action<int>> curriedAdd2Args = add2Args.Curry();

curriedAdd2Args(1)(2);

// (int, int, int) -> void

Action<int, int, int> add3Args = (a, b, c) => (a + b + c).WriteLine();

add2Args(1, 2);

// int -> int -> int -> void

Func<int, Func<int, Action<int>>> curriedAdd3Args = add3Args.Curry();

curriedAdd3Args(1)(2)(3);

}

Uncurry function

The opposite transformation from a sequence of nested single parameter functions to a function with multiple parameters is called uncurrying. Uncurry extension methods implemented for higher-order functions with a chain of calls:

// Transform T1 -> T2 ->TResult

// to (T1, T2) -> TResult.

public static Func<T1, T2, TResult> Uncurry<T1, T2, TResult>(

this Func<T1, Func<T2, TResult>> function) =>

(value1, value2) => function(value1)(value2);

// Transform T1 -> T2 ->T3 -> TResult

// to (T1, T2, T3) -> TResult.

public static Func<T1, T2, T3, TResult> Uncurry<T1, T2, T3, TResult>(

this Func<T1, Func<T2, Func<T3, TResult>>> function) =>

(value1, value2, value3) => function(value1)(value2)(value3);

// Transform T1 -> T2 ->T3 -> T4 -> TResult

// to (T1, T2, T3, T4) ->TResult.

public static Func<T1, T2, T3, T4, TResult> Uncurry<T1, T2, T3, T4, TResult>(

this Func<T1, Func<T2, Func<T3, Func<T4, TResult>>>> function) =>

(value1, value2, value3, value4) => function(value1)(value2)(value3)(value4);

// ...

// Transform T1 -> T2 ->void

// to (T1, T2) -> void.

public static Action<T1, T2> Uncurry<T1, T2>(

this Func<T1, Action<T2>> function) => (value1, value2) =>

function(value1)(value2);

// Transform T1 -> T2 ->T3 -> void

// to (T1, T2, T3) -> void.

public static Action<T1, T2, T3> Uncurry<T1, T2, T3>(

this Func<T1, Func<T2, Action<T3>>> function) =>

(value1, value2, value3) => function(value1)(value2)(value3);

// Transform T1 -> T2 ->T3 -> T4 -> void

// to (T1, T2, T3, T4) ->void.

public static Action<T1, T2, T3, T4> Uncurry<T1, T2, T3, T4>(

this Func<T1, Func<T2, Func<T3, Action<T4>>>> function) =>

(value1, value2, value3, value4) => function(value1)(value2)(value3)(value4);

// ...

The usage of Uncurry is also straightforward:

internal static void Uncurry()

// int ->int -> int

Func<int, Func<int, int>> curriedAdd2Args = a => b => a + b;

// (int ->int) -> int

Func<int, int, int> add2Args = curriedAdd2Args.Uncurry();

int add2ArgsResult = add2Args(1, 2);

// int ->int -> int -> void

Func<int, Func<int, Action<int>>> curriedAdd3Args = a => b => c => (a + b + c).WriteLine();

// (int ->int -> int) -> void

Action<int, int, int> add3Args = curriedAdd3Args.Uncurry();

add3Args(1, 2, 3);

}

Partially apply function

Another function transformation similar to function currying is function partial application. Function application is just another word for function call. For function with multiple parameters, function partial application means to call that function with partial arguments (typically, a single argument) instead of all arguments. Similar to currying, the implementation of function partial application also involves higher-order function and closure. The difference is, partial application does not result a single function with fewer parameters, not a sequence of nested single parameter functions. For example, a first-order function of type (int, int) -> int can be partially applied with 1 argument and transformed to another first-order function of type int ->int, a first-order function of type (int, int, int) -> void can be partially applied with 1 argument and transformed to another first-order function of type (int, int) -> void:

internal static void FirstOrderToFirstOrder()

// (int, int) -> int

Func<int, int, int> add2Args = (a, b) => a + b;

// int -> int

Func<int, int> add1ArgAnd1Variable = b => 1 + b; // Partially apply add2Args with a = 1.

// (int, int, int) -> void

Action<int, int, int> add3Args = (a, b, c) => (a + b + c).WriteLine();

add2Args(1, 2);

// (int, int) -> void

Action<int, int> add2ArgsAnd1Variable = (b, c) => (1 + b + c).WriteLine(); // Partially apply add2Args with a = 1.

// add2ArgsAnd1Variable can be called with 2 arguments, or be partially applied again with b = 2:

// int -> void

Action<int> add1ArgsAnd2Variable = c => (1 + 2 + c).WriteLine();

}

Generally, following Partial extension methods for Func and Action generic delegate types are higher-order functions. When Partial is used for a function with 2 or more parameters, it accepts the first parameter of that function, and outputs another function that accepts the rest of parameters of the original function:

// Input: function of type (T1, T2) -> TResult, first parameter of type T1.

// Output: function of type T2 -> TResult.

public static Func<T2, TResult> Partial<T1, T2, TResult>(

this Func<T1, T2, TResult> function, T1 value1) =>

value2 => function(value1, value2);

// Input: function of type (T1, T2, T3) -> TResult, first parameter of type T1.

// Output: function of type (T2, T3) -> TResult.

public static Func<T2, T3, TResult> Partial<T1, T2, T3, TResult>(

this Func<T1, T2, T3, TResult> function, T1 value1) =>

(value2, value3) => function(value1, value2, value3);

// Input: function of type (T1, T2, T3, T4) -> TResult, first parameter of type T1.

// Output: function of type (T2, T3, T4) -> TResult.

public static Func<T2, T3, T4, TResult> Partial<T1, T2, T3, T4, TResult>(

this Func<T1, T2, T3, T4, TResult> function, T1 value1) =>

(value2, value3, value4) => function(value1, value2, value3, value4);

// ...

// Input: function of type (T1, T2) -> void, first parameter of type T1.

// Output: function of type T2 -> void.

public static Action<T2> Partial<T1, T2>(

this Action<T1, T2> function, T1 value1) =>

value2 => function(value1, value2);

// Input: function of type (T1, T2, T3) -> void, first parameter of type T1.

// Output: function of type (T2, T3) -> void.

public static Action<T2, T3> Partial<T1, T2, T3>(

this Action<T1, T2, T3> function, T1 value1) =>

(value2, value3) => function(value1, value2, value3);

// Input: function of type (T1, T2, T3, T4) -> void, first parameter of type T1.

// Output: function of type (T2, T3, T4) -> void.

public static Action<T2, T3, T4> Partial<T1, T2, T3, T4>(

this Action<T1, T2, T3, T4> function, T1 value1) =>

(value2, value3, value4) => function(value1, value2, value3, value4);

// ...

The following example demonstrates how to call Partial for partial application of functions with multiple parameters:

internal static void PartiallyApply()

// (int, int) -> int

Func<int, int, int> add2Args = (a, b) => a + b;

// int -> int

Func<int, int> add1ArgAnd1Variable = add2Args.Partial(1);

int add1ArgAnd1VariableResult = add1ArgAnd1Variable(2);

// (int, int, int) -> void

Action<int, int, int> add3Args = (a, b, c) => (a + b + c).WriteLine();

// (int, int) -> void

Action<int, int> add2ArgsAnd1Variable = add3Args.Partial(1);

add2ArgsAnd1Variable(2, 3);

}

For curried function, partial application means a chain of calls with fewer arguments than total:

internal static void PartiallyApplyCurriedFunction()

// int -> int -> int

Func<int, Func<int, int>> curriedAdd2Args = a => b => a + b;

// int -> int

Func<int, int> partiallyAppliedCurriedAdd2Args = curriedAdd2Args(1);

// int -> int -> int -> void

Func<int, Func<int, Action<int>>> curriedAdd3Args = a => b => c => (a + b + c).WriteLine();

// int -> void

Action<int> partiallyAppliedCurriedAdd3Args = curriedAdd3Args(1)(2);

}

First-class function

Higher-order function enables function to be function’s input and output, which is an important aspect of be first-class citizenship in language. First-class function means in C# function supports all the generally available operations for other entities like object. These operations include being a variable, being a type’s field, being a function’s input, being a function’s output, being equality testable, etc. This can be demonstrated by comparing C# function with C# object side by side. First of all, with delegate type and delegate instance, function and object both have type and instance, and an instance can be variable, can have alias and immutable alias:

internal static void Object()

Data value = new Data(0);

ref Data alias = ref value;

ref readonly Data immutableAlias = ref value;

internal static void Function()

Function value1 = Function; // Named function.

Function value2 = () => { }; // Anonymous function.

ref Function alias = ref value1;

ref readonly Function immutableAlias = ref value2;

}

With delegate type and delegate instance, function and object can both be stored with static and instance field of type:

internal class Fields

private static Data staticDataField = new Data(0);

private static Function staticNamedFunctionField = Function;

private static Function staticAnonymousFunctionField = () => { };

private Data instanceDataField = new Data(0);

private Function instanceNamedFunctionField = Function;

private Function instanceAnonymousFunctionField = () => { };

}

With local function and lambda expression, function and object can both be nested:

internal partial class Data

internal Data Inner { get; set; }

internal static void NestedObject()

Data outer = new Data(1)

Inner = new Data(2)

};

internal static void NestedFunction()

void Outer()

void Inner() { }

Function outer = () =>

Function inner = () => { };

};

}

With closure to capture free variable, function and object can both access data out of the scope:

internal class OuterClass

const int Outer = 1;

class InnerClass

const int Inner = 2;

int sum = Inner + Outer;

internal static void OuterFunction()

const int Outer = 1;

void InnerFunction()

const int Inner = 2;

int sum = Inner + Outer;

new Function(() =>

const int Inner = 2;

int sum = Inner + Outer;

})();

}

With higher-order function, function and object can both be input and output of function:

internal static Data Function(Data value) => value;

internal static Function Function(Function value) => value;

With delegate type and delegate instance, function and object can both be equality testable:

internal partial class Data

public override bool Equals(object obj) =>

object.ReferenceEquals(this, obj) || this.Value == (obj as Data)?.Value;

public override int GetHashCode() => this.Value.GetHashCode();

public static bool operator ==(Data data1, Data data2) => data1?.Value == data2?.Value;

public static bool operator !=(Data data1, Data data2) => !(data1 == data2);

internal static void ObjectEquality()

Data value1 = new Data(1);

Data value2 = new Data(1);

object.ReferenceEquals(value1, value2).WriteLine(); // False

object.Equals(value1, value2).WriteLine(); // True

value1.Equals(value2).WriteLine(); // True

(value1 == value2).WriteLine(); // True

(value1.GetHashCode() == value2.GetHashCode()).WriteLine(); // True.

EqualityComparer<Data>.Default.Equals(value1, value2).WriteLine(); // True

internal static void FunctionEquality()

Function value1 = Function;

Function value2 = Function;

object.ReferenceEquals(value1, value2).WriteLine(); // False

object.Equals(value1, value2).WriteLine(); // True

value1.Equals(value2).WriteLine(); // True

(value1 == value2).WriteLine(); // True

(value1.GetHashCode() == value2.GetHashCode()).WriteLine(); // True.

EqualityComparer<Function>.Default.Equals(value1, value2).WriteLine(); // True

}

Here Data type overrides object type’s Equals method as well as the == and != operators, so that, 2 Data instances considered logically equal if they encapsulate the same int value. It also overrides object type’s GetHashCode, so that 2 equal Data instances have the same hash code. If 2 variables are 2 Data instances encapsulating the same int value, apparently these 2 variables are not reference equal, but logically equal with the same hash code. C# provides these equality APIs through System.Delegate and System.MulticastDelegate. So similarly, if 2 variables are 2 delegate instances representing the same function, they are not reference equal, but logically equal with the same hash code as well.

So, with rich functional features including delegate, local function, lambda expression, closure and higher-order function, C#’s named function and anonymous function are first-class citizens, and C# is a functional language. Besides these aspects, C# functions can be composed just like object composition. Function composition is discussed in the next chapter.

Summary

This chapter discussed higher-order function, and its related concepts, including first-order function, currying function, uncurrying function, partially applying function, and lambda operator’s associativity. Higher-order function works with other C# features, and enables first-class functions in C#.

C# Functional Programming In-Depth (7) Expression Tree: Function as Data

Fri, 07 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

Lambda expression as expression tree

C# lambda expression is a powerful syntactic sugar. Besides representing anonymous function, the same syntax can also represent expression tree. An expression tree is an immutable tree data structure that represents structure of code. For example:

internal static void ExpressionLambda()

Expression<Func<int, bool>> isPositiveExpression = int32 => int32 > 0;

// Compare to: Func<int, bool> isPositive = int32 => int32 > 0;

}

This time, the expected type for the lambda expression is no longer a Func<int, bool> function type, but Expression<Func<int, bool>> data type. The lambda expression here is no longer compiled to executable anonymous function code, but an expression tree data structure representing that function’s code logic.

Metaprogramming: function as abstract syntax tree

The above lambda expression has exactly the same syntax as anonymous function, but it is compiled to code that builds expression tree:

internal static void CompiledExpressionLambda()

ParameterExpression parameterExpression = Expression.Parameter(type: typeof(int), name: "int32"); // int32 parameter.

ConstantExpression constantExpression = Expression.Constant(value: 0, type: typeof(int)); // 0

BinaryExpression greaterThanExpression = Expression.GreaterThan(

left: parameterExpression, right: constantExpression); // int32 > 0

Expression<Func<int, bool>> isPositiveExpression = Expression.Lambda<Func<int, bool>>(

body: greaterThanExpression, // ... => int32 > 0

parameters: parameterExpression); // int32 => ...

}

Here the Expression<Func<int bool>> instance represents the entire tree, the ParameterExpression, ConstantExpression, BinaryExpression instances are nodes in that tree. And they are all derived from System.Linq.Expressions.Expression type:

namespace System.Linq.Expressions

public abstract class Expression

public virtual ExpressionType NodeType { get; }

public virtual Type Type { get; }

// Other members.

public class ParameterExpression : Expression

public string Name { get; }

// Other members.

public class ConstantExpression : Expression

public object Value { get; }

// Other members.

public class BinaryExpression : Expression

public Expression Left { get; }

public Expression Right { get; }

// Other members.

public abstract class LambdaExpression : Expression

public Expression Body { get; }

public ReadOnlyCollection<ParameterExpression> Parameters { get; }

// Other members.

public sealed class Expression<TDelegate> : LambdaExpression

public TDelegate Compile();

// Other members.

}

The above expression tree data structure can be visualized as:

Expression<Func<int, bool>> (NodeType = Lambda, Type = Func<int, bool>)

|_Parameters

| |_ParameterExpression (NodeType = Parameter, Type = int)

| |_Name = "int32"

|_Body

|_BinaryExpression (NodeType = GreaterThan, Type = bool)

|_Left

| |_ParameterExpression (NodeType = Parameter, Type = int)

| |_Name = "int32"

|_Right

|_ConstantExpression (NodeType = Constant, Type = int)

|_Value = 0

So, this expression tree is an abstract syntax tree (AST), representing the abstract syntactic structure of C# function (int32 => int32 > 0)’s source code. Notice each node has NodeType property and Type property. NodeType returns the syntax type in the tree, and Type returns the represented .NET type. For example, above ParameterExpression represents parameter node of int type in the source code, so its NodeType is Parameter and its Type is int.

To summarize, the differences between

Func<int, bool>isPositive = int32 => int32 > 0;// Code.

and

Expression<Func<int, bool>> isPositiveExpression = int32 => int32> 0;// Data.

are:

· isPositive variable is a function represented by delegate instance, and can be called. The lambda expression int32 => int32 > 0 is compiled to executable code. When isPositive is called, this code is executed.

· isPositiveExpression variable is an abstract syntax tree data structure. So apparently it cannot be called like an executable function. The lambda expression int32 => int32 > 0 is compiled to the building of an expression tree, where each node is an Expression instance. This entire tree represents the syntactic structure and logic of function int32 => int32 > 0. This tree’s top node is an Expression<Func<int, bool>> instance, since this is a lambda expression. It has 2 child nodes:

o A Parameters node which is a ParameterExpression collection, representing all the parameters of the lambda expression. The above lambda expression has 1 parameter, so this collection contains one node:

§ A ParameterExpression instance, representing the int parameter named “int32”.

o A Body node representing the lambda expression’s body, which is a BinaryExpression instance, representing the body is a “>” (greater than) comparison operation of 2 operands. So it has 2 child nodes:

§ Its Left child node is a reference of above ParameterExpression instance, representing the left operand of > operator.

§ Its Right child node is a ConstantExpression instance with value 0, representing the right operand of > operator.

Automatically generating expression tree from function-like code provides great convenience for metaprogramming in C#. As mentioned in the introduction chapter, metaprogramming is to generate or manipulate program code as data. With the generated expression tree, since each node is strong typed with rich information, the nodes can be traversed to obtain the information of the represented function’s C# source code, and process the information, like generating code of the same logic in another language. Here isPositiveExpression represents the function logic to predicate whether an int value is greater than a constant 0, and it can be used to generate equivalent CIL code, or a WHERE clause with greater-than-0 predicate in SQL code, etc.

.NET expressions

Besides above ParameterExpression, ConstantExpression, BinaryExpression, LambdaExpression, .NET Standard provides a rich collection of expressions nodes. The following is their inheritance hierarchy:

· Expression

o BinaryExpression

o BlockExpression

o ConditionalExpression

o ConstantExpression

o DebugInfoExpression

o DefaultExpression

o DynamicExpression

o GotoExpression

o IndexExpression

o InvocationExpression

o LabelExpression

o LambdaExpression

§ Expression<TDelegate>

o ListInitExpression

o LoopExpression

o MemberExpression

o MemberInitExpression

o MethodCallExpression

o NewArrayExpression

o NewExpression

o ParameterExpression

o RuntimeVariablesExpression

o SwitchExpression

o TryExpression

o TypeBinaryExpression

o UnaryExpression

And, as demonstrated above, expression can be instantiated by calling the factory methods of Expression type:

public abstract partial class Expression

public static ParameterExpression Parameter(Type type, string name);

public static ConstantExpression Constant(object value, Type type);

public static BinaryExpression GreaterThan(Expression left, Expression right);

public static Expression<TDelegate> Lambda<TDelegate>(Expression body, params ParameterExpression[] parameters);

}

Expression has many other factory methods to cover all the expression instantiation cases:

public abstract partial class Expression

public static BinaryExpression Add(Expression left, Expression right);

public static BinaryExpression Subtract(Expression left, Expression right);

public static BinaryExpression Multiply(Expression left, Expression right);

public static BinaryExpression Divide(Expression left, Expression right);

public static BinaryExpression Equal(Expression left, Expression right);

public static UnaryExpression ArrayLength(Expression array);

public static UnaryExpression Not(Expression expression);

public static ConditionalExpression Condition(Expression test, Expression ifTrue, Expression ifFalse);

public static NewExpression New(ConstructorInfo constructor, params Expression[] arguments);

public static MethodCallExpression Call(MethodInfo method, params Expression[] arguments);

public static BlockExpression Block(params Expression[] expressions);

// Other members.

}

One expression node type can have different possible NodeType values. For example:

· UnaryExpression represents any unary operation with an operator and an operand. Its NodeType can be ArrayLength, Negate, Not, Convert, Decrement, Increment, Throw, UnaryPlus, etc.

· BinaryExpression represents any binary operation with an operator, a left operand, and a right operand, its NodeType can be Add, And, Assign, Divide, Equal, .GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual, Modulo, Multiply, NotEqual, Or, Power, Subtract, etc.

So far C# compiler only implements this “function as data” syntactic sugar for expression lambda, and it is not available to statement lambda yet. The following code cannot be compiled:

internal static void StatementLambda()

Expression<Func<int, bool>>isPositiveExpression = int32 =>

Console.WriteLine(int32);

return int32 > 0;

}; // Cannot be compiled.

}

The above expression tree has to be built manually:

internal static void StatementLambda()

ParameterExpression parameterExpression = Expression.Parameter(type: typeof(int), name: "int32"); // int32 parameter.

Expression<Func<int, bool>> isPositiveExpression = Expression.Lambda<Func<int, bool>>(

body: Expression.Block( // ... => {

// Console.WriteLine(int32);

arg0: Expression.Call(method: new Action<int>(Console.WriteLine).Method, arg0: parameterExpression),

// return int32 > 0;

arg1: Expression.GreaterThan(left: parameterExpression, right: Expression.Constant(value: 0, type: typeof(int)))), // }

parameters: parameterExpression); // int32 => ...

}

Compile expression tree to CIL

Expression tree is not executable code but data - abstract syntax tree. In C# and LINQ, expression tree is usually used to represent the abstract syntactic structure of function’s source code, so that it can be compiled or translated to other domain-specific languages, like SQL query, HTTP request, etc. To demonstrate this, take the following simple mathematics function as example, which accepts double parameters, then executes the 4 basic binary arithmetical calculation: add, subtract, multiply, divide, and returns double return:

internal static void Infix()

Expression<Func<double, double, double, double, double, double>>expression =

(a, b, c, d, e) => a + b - c * d / 2D + e * 3D;

}

The entire tree can be visualized as:

Expression<Func<double, double, double, double, double, double>> (NodeType = Lambda, Type = Func<double, double, double, double, double, double>)

|_Parameters

| |_ParameterExpression (NodeType = Parameter, Type = double)

| | |_Name = "a"

| |_ParameterExpression (NodeType = Parameter, Type = double)

| | |_Name = "b"

| |_ParameterExpression (NodeType = Parameter, Type = double)

| | |_Name = "c"

| |_ParameterExpression (NodeType = Parameter, Type = double)

| | |_Name = "d"

| |_ParameterExpression (NodeType = Parameter, Type = double)

| |_Name = "e"

|_Body

|_BinaryExpression (NodeType = Add, Type = double)

|_Left

| |_BinaryExpression (NodeType = Subtract, Type = double)

| |_Left

| | |_BinaryExpression (NodeType = Add, Type = double)

| | |_Left

| | | |_ParameterExpression (NodeType = Parameter, Type = double)

| | | |_Name = "a"

| | |_Right

| | |_ParameterExpression (NodeType = Parameter, Type = double)

| | |_Name = "b"

| |_Right

| |_BinaryExpression (NodeType = Divide, Type = double)

| |_Left

| | |_BinaryExpression (NodeType = Multiply, Type = double)

| | |_Left

| | | |_ParameterExpression (NodeType = Parameter, Type = double)

| | | |_Name = "c"

| | |_right

| | |_ParameterExpression (NodeType = Parameter, Type = double)

| | |_Name = "d"

| |_Right

| |_ConstantExpression (NodeType = Constant, Type = double)

| |_Value = 2

|_Right

|_BinaryExpression (NodeType = Multiply, Type = double)

|_Left

| |_ParameterExpression (NodeType = Parameter, Type = double)

| |_Name = "e"

|_Right

|_ConstantExpression (NodeType = Constant, Type = double)

|_Value = 3

This is a very simple syntax tree, where:

· each internal node is a binary node (BinaryExpression instance) representing add, subtract, multiply, or divide binary operations;

· each leaf node is either a parameter (ParameterExpression instance), or a constant (ConstantExpression instance).

In total there are 6 kinds of nodes in this tree:

· add: BinaryExpression { NodeType = ExpressionType.Add }

· subtract: BinaryExpression { NodeType = ExpressionType.Subtract }

· multiply: BinaryExpression { NodeType = ExpressionType.Multiply }

· divide: BinaryExpression { NodeType = ExpressionType.Divide}

· constant: ParameterExpression { NodeType = ExpressionType.Constant }

· parameter: ConstantExpression { NodeType = ExpressionType.Parameter }

Traverse expression tree

The above expression is an infix expression, where each operator is in the middle of its 2 operands. It is very easy to traverse the expression tree and convert it to a prefix form (also called Polish notation), where each operator precedes its operands, just like a function to be called:

internal static string PreOrderOutput(this LambdaExpression expression)

string VisitNode(Expression node)

switch (node.NodeType)

case ExpressionType.Add:

case ExpressionType.Subtract:

case ExpressionType.Multiply:

case ExpressionType.Divide:

BinaryExpression binary = (BinaryExpression)node;

// Pre-order output: current node, left child, right child.

return $"{binary.NodeType}({VisitNode(binary.Left)}, {VisitNode(binary.Right)})";

case ExpressionType.Constant:

return ((ConstantExpression)node).Value.ToString();

case ExpressionType.Parameter:

return ((ParameterExpression)node).Name;

default:

throw new ArgumentOutOfRangeException(nameof(expression));

return VisitNode(expression.Body);

}

It calls a local function to recursively visit each node in lambda expression’s body. If the current node is a BinaryExpression, it recursively visits the Left and Right child nodes, and outputs string representation in pre-order: Operator(Left, Right). If the current node is a ConstantExpression or ParameterExpression, it output the string representation of current node, and terminates the recursion. The above function can convert the infix expression to a prefix expression in function call style:

internal static void Prefix()

Expression<Func<double, double, double, double, double, double>> infix =

(a, b, c, d, e) => a + b - c * d / 2D + e * 3D;

string prefix = infix.PreOrderOutput();

prefix.WriteLine(); // Add(Subtract(Add(a, b), Divide(Multiply(c, d), 2)), Multiply(e, 3))

}

Actually, .NET Standard provides a built-in System.Linq.Expressions.ExpressionVisitor type with an object-oriented design to traverse general expression trees. This book traverses expression tree in functional paradigm because it is very intuitive for simple arithmetic expression with small code.

Expression tree to CIL at runtime

If the output is in post-order, where operands are followed by operator, then it can be viewed as the stack operations: push operands to stack, then execute the operator with the operands and push the result on the stack. This is how the evaluation stack based CIL language works. So, a different traversal output in post-order can represent executable CIL instructions. An CIL instruction can be represented by System.Reflection.Emit.OpCode structures with optional argument. So, the output can be a list of instruction-argument pairs (Here a pair is represented by a 2-tuple of OpCode and object. C#’s tuple is discussed in detail in the immutability chapter):

internal static List<(OpCode, object)> PostOrderOutput(this LambdaExpression expression)

List<(OpCode, object)> VisitNode(Expression node)

switch (node.NodeType)

case ExpressionType.Add:

return VisitBinary((BinaryExpression)node, OpCodes.Add);

case ExpressionType.Subtract:

return VisitBinary((BinaryExpression)node, OpCodes.Sub);

case ExpressionType.Multiply:

return VisitBinary((BinaryExpression)node, OpCodes.Mul);

case ExpressionType.Divide:

return VisitBinary((BinaryExpression)node, OpCodes.Div);

case ExpressionType.Constant:

return new List<(OpCode, object)>()

(OpCodes.Ldc_R8, ((ConstantExpression)node).Value) // Push constant to stack.

};

case ExpressionType.Parameter:

int parameterIndex = expression.Parameters.IndexOf((ParameterExpression)node);

return new List<(OpCode, object)>()

(OpCodes.Ldarg_S, parameterIndex) // Push parameter of the specified index to stack.

};

default:

throw new ArgumentOutOfRangeException(nameof(expression));

List<(OpCode, object)> VisitBinary(BinaryExpression binary, OpCode postfix)

// Post-order output: left child, right child, current node.

List<(OpCode, object)> instructions = VisitNode(binary.Left);

instructions.AddRange(VisitNode(binary.Right));

instructions.Add((postfix, null)); // Operate and push the result to stack.

return instructions;

return VisitNode(expression.Body);

}

The following code outputs a list of converted CIL instruction:

internal static void Postfix()

Expression<Func<double, double, double, double, double, double>> infix =

(a, b, c, d, e) => a + b - c * d / 2D + e * 3D;

List<(OpCode, object)> postfix = infix.PostOrderOutput();

foreach ((OpCode instruction, object argument) in postfix)

$"{instruction} {argument}".WriteLine();

// ldarg.s 0

// ldarg.s 1

// add

// ldarg.s 2

// ldarg.s 3

// mul

// ldc.r8 2

// div

// sub

// ldarg.s 4

// ldc.r8 3

// mul

// add

}

So, the source code in C# language represented by this expression tree is successfully converted to instructions CIL language. In another word, C# is compiled to CIL with the help of expression tree.

Expression tree to function at runtime

The above compiled CIL code is executable. With C#’s metaprogramming capability, a function can be generated at runtime (instead of compiled time), where the compiled CIL code can be emitted into:

internal static TDelegate CompileToCil<TDelegate>(this Expression<TDelegate> expression)

DynamicMethod dynamicFunction = new DynamicMethod(

name: string.Empty,

returnType: expression.ReturnType,

parameterTypes: expression.Parameters.Select(parameter => parameter.Type).ToArray(),

m: MethodBase.GetCurrentMethod().Module);

EmitCil(dynamicFunction.GetILGenerator(), expression.PostOrderOutput());

return (TDelegate)(object)dynamicFunction.CreateDelegate(typeof(TDelegate));

void EmitCil(ILGenerator generator, List<(OpCode, object)> cil)

foreach ((OpCode instruction, object argument) in cil)

if (argument == null)

generator.Emit(instruction); // add, sub, mul, div has no argument.

else if (argument is int)

generator.Emit(instruction, (int)argument); // ldarg.s has int argument of parameter index.

else if (argument is double)

generator.Emit(instruction, (double)argument); // ldc.r8 has double argument of constant.

generator.Emit(OpCodes.Ret); // Return the result.

}

The following code demonstrate how to use it to compile C# expression tree of C# code into CIL code encapsulated by a function, then call the function to execute the compiled CIL code, and finally get the result:

internal static void CompileAndRun()

Expression<Func<double, double, double, double, double, double>> expression =

(a, b, c, d, e) => a + b - c * d / 2D + e * 3D;

Func<double, double, double, double, double, double> function = expression.CompileToCil();

double result = function(1D, 2D, 3D, 4D, 5D);

result.WriteLine(); // 12

}

.NET provides a built-in API, System.Linq.Expressions.Expression<TDelegate>’s Compile method, for this purpose - compile expression tree to executable function at runtime:

internal static void BuiltInCompile()

Expression<Func<double, double, double, double, double, double>>infix =

(a, b, c, d, e) => a + b - c * d / 2D + e * 3D;

Func<double, double, double, double, double, double> function = infix.Compile();

double result = function(1D, 2D, 3D, 4D, 5D

}

Internally, Expression<TDelegate>.Compile calls APIs of System.Linq.Expressions.Compiler.LambdaCompile, which is a complete compiler implementation for general expression tree to CIL (not just for simple expression in the above example).

Lambda expression and LINQ query

As such a powerful syntactic sugar, lambda expression is very important in LINQ query. In local LINQ query for data in memory, lambda expression is compiled as anonymous function. In remote LINQ query for data in a different data domain, lambda expression is compiled as expression tree, which can be compiled or translated from C# language to that domain’s specific language. In above examples, expression tree is compiled to executable CIL. In other scenarios of LINQ remote query, like LINQ to Entities, expression tree can be translated to a part of SQL query that can be executed in database The following examples are a typical LINQ to Objects query (for local data in-memory) and a similar LINQ to Entities query (for remote data in database):

internal static void LocalLinqToObjectsQuery(IEnumerable<Product> source) // Get source.

IEnumerable<Product> query = source.Where(product => product.ListPrice > 0M); // Define query.

foreach (Product result in query) // Execute query.

result.Name.WriteLine();

internal static void RemoteLinqToEntitiesQuery(IQueryable<Product> source) // Get source.

IQueryable<Product> query = source.Where(product => product.ListPrice > 0M); // Define query.

foreach (Product result in query) // Execute query.

result.Name.WriteLine();

}

As mentioned in the introduction chapter, local LINQ query is represented by IEnumerable<T>, and remote LINQ query is represented by IQueryable<T>. They have different LINQ query methods, take above Where as example:

namespace System.Linq

public static class Enumerable

public static IEnumerable<TSource> Where<TSource>(

this IEnumerable<TSource> source, Func<TSource, bool> predicate);

public static class Queryable

public static IQueryable<TSource> Where<TSource>(

this IQueryable<TSource> source, Expression<Func<TSource, bool>> predicate);

}

Where query filters the local/remote data source with the specified predicate as anonymous function/expression tree. For each value in the data source, if the predicate evaluates to true, the data value should be in the query results; otherwise, the data value should be ignored. In the above LINQ to Object/LINQ to Entities queries, the Where query and predicate has exactly the same logic and syntax (filter the source of products, and only keep the products with list price greater than 0), but they are compiled totally differently:

internal static void EquivalentLinqToObjectsQuery(IEnumerable<Product> source) // Get source.

Func<Product, bool> predicateFunction = product => product.ListPrice > 0M;

// Compiled to named function with cache field.

IEnumerable<Product> query = Enumerable.Where(source, predicateFunction); // Define query.

foreach (Product result in query) // Execute query.

result.Name.WriteLine();

internal static void EquivalentLinqToEntitiesQuery(IQueryable<Product> source) // Get source.

Expression<Func<Product, bool>> predicateExpression = product => product.ListPrice > 0M;

// Compiled to:

// ParameterExpression productParameter = Expression.Parameter(type: typeof(Product), name: "product");

// Expression<Func<Product, bool>> predicateExpression = Expression.Lambda<Func<Product, bool>>(

// body: Expression.GreaterThan(

// left: Expression.Property(expression: productParameter, propertyName: nameof(Product.ListPrice)),

// right: Expression.Constant(value: 0M, type: typeof(decimal))),

// productParameter);

IQueryable<Product> query = Queryable.Where(source, predicateExpression); // Define query.

foreach (Product result in query) // Execute query.

result.Name.WriteLine();

}

At runtime, when the local query executes, the anonymous function is called to predict each local value in the local data source, and the remote query is translated to a WHERE clause in SQL query, then submit to the remote data source and execute. The local query’s usage, execution, and implementation is discussed in the LINQ to Objects chapters. The remote query’s usage, execution and the translation from expression tree to SQL will be discussed in LINQ to Entities chapters.

Summary

This chapter discusses C#’s lambda expression, a very powerful syntactic sugar that is very useful in functional programming and LINQ. Lambda expression can be used to define anonymous function. C# implements this by compiling lambda expression to named function. C# also implements lambda expression’s expression body syntax for all kinds of named functions. Lambda expression can also be used to build expression tree, which is a data structure of abstract syntax tree. This chapter also demonstrates how to traverse expression tree and convert C# code to another language, as well as how LINQ implement local query and remote query with one syntax for different query methods and lambda expressions.

C# Functional Programming In-Depth (6) Anonymous Function and Lambda Expression

Thu, 06 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

Besides named function, C# also supports anonymous function, represented by anonymous method or lambda expression with no function name at design time. Lambda expression can also represent expression tree, This chapter discusses lambda expression as a functional feature of C# language. Lambda expression is also the core concept of lambda calculus, where functional programming originates. It is revisited in the Lambda Calculus chapters.

Anonymous method

As discussed in the delegate chapter, a function can be represented by a delegate instance, and delegate instance can be initialized with a named function:

internal static bool IsPositive(int int32) { return int32 > 0; }

internal static void NamedFunction()

Func<int, bool> isPositive = IsPositive;

bool result = isPositive(0);

}

C# 2.0 introduces a syntactic sugar called anonymous method, enabling function to be defined inline with the delegate keyword. The above function can be inline as:

internal static void AnonymousMethod()

Func<int, bool> isPositive = delegate (int int32) { return int32 > 0; };

bool result = isPositive(0);

}

This time delegate instance is initialized with an anonymous function, which does not have function name at design time. At compile time, a named function is generated. The above example is compiled to:

[CompilerGenerated]

private static Func<int, bool> cachedIsPositive;

[CompilerGenerated]

private static bool CompiledIsPositive(int int32) { return int32 > 0; }

internal static void CompiledAnonymousMethod()

Func<int, bool> isPositive;

if (cachedIsPositive == null)

cachedIsPositive = new Func<int, bool>(CompiledIsPositive);

isPositive = cachedIsPositive;

bool result = isPositive.Invoke(0);

}

Besides named function, C# compiler also generates a cache field for performance. As discussed in the delegate chapter, a delegate instance is actually an object with an Invoke method. When AnonymousMethod is called for the first time, the delegate instance is constructed with the generated named function, and stored in the static cache fieled forever. When AnonymousMethod is called again, the cached delegate instance is reused.

Lambda expression as anonymous function

C# 3.0 introduces lambda expression syntactic sugar, which can define anonymous function with a lambda operator instead of delegate keyword:

internal static void AnonymousFunction()

Func<int, bool> isPositive = (int int32) => { return int32 > 0; };

// Equivalent to: Func<int, bool> isPositive = int32 => int32 > 0;

bool result = isPositive(0);

}

Its compilation is identical to above anonymous method example. The => operator is called lambda operator and reads “go to”. Lambda expression can be further shortened:

· if the type of parameter can be inferred (for example, from the specified function type), the type of parameter can be omitted. In above example, the lambda expression’s parameter type can be inferred to be int from the provided function type int –> bool, so it can be simplified to (int32) => { return int32 > 0; }.

· if lambda expression has one parameter, the parentheses for the parameter can be omitted. So, the above lambda expression can be simplified to int32 => { return int32 > 0; }.

· if the body of the lambda expression has only one statement, the curly brackets for the body and the return keyword can be omitted, which is call expression body syntax. So, the above lambda expression can be just int32 => int32 > 0.

Lambda expression with expression body are called expression lambda, for example:

internal static void ExpressionLambda()

Func<int, int, int> add = (int32A, int32B) => int32A + int32B;

Func<int, bool>isPositive = int32 => int32 > 0;

Action<int> traceLine = int32 => int32.WriteLine();

}

When a lambda expression having more than one statements in the body, its body has to be a block with curly brackets. It is called statement lambda:

internal static void StatementLambda()

Func<int, int, int> add = (int32A, int32B) =>

int sum = int32A + int32B;

return sum;

};

Func<int, bool>isPositive = int32 =>

int32.WriteLine();

return int32> 0;

};

Action<int> traceLine = int32 =>

int32.WriteLine();

Trace.Flush();

};

}

For delegate instantiation, lambda expression (both expression lambda and statement lambda) can also be used with the constructor call syntax and type conversion syntax, which is similar to using named function discussed in the delegate chapter:

internal static void Constructor()

Func<int, bool> isPositive = new Func<int, bool>(int32 => int32 > 0);

bool result = isPositive(0);

internal static void Conversion()

Func<int, bool> isPositive = (Func<int, bool>)(int32 => int32 > 0);

bool result = isPositive(0);

internal static void Simplified()

Func<int, bool> isPositive = int32 => int32 > 0;

bool result = isPositive(0);

}

These 3 syntaxes are compiled identically.

Immediately-invoked function expression

An anonymous function is not required to be assigned to a function variable. It can be used directly. The following example directly calls an anonymous function definition. Unfortunately, it cannot be compiled:

internal static void CallAnonymousFunction(int arg)

(int32 => int32 > 0)(arg); // Cannot be compiled..

}

The reason is, C# compiler cannot inter the type information of the lambda expression. The type information can be provided with the constructor call or type conversion syntax above code cannot be compiled because C# compiler cannot infer any type for the lambda expression. For this kind of IIFE (), the above constructor call syntax, or type conversion syntax can be used to provide type information to compiler:

internal static void CallLambdaExpressionWithConstructor(int arg)

bool result = new Func<int, bool>(int32 => int32 > 0)(arg);

internal static void CallLambdaExpressionWithConversion(int arg)

bool result = ((Func<int, bool>)(int32 => int32 > 0))(arg);

}

In functional programming, this pattern is called immediately-invoked function expression (IIFE). There is no function name or function variable (delegate instance) name involved at design time. At compile time, C# compiler generates identical code of named function for the above 2 syntaxes:

[CompilerGenerated]

[Serializable]

private sealed class Functions

public static readonly Functions Singleton = new Functions();

public static Func<int, bool> cachedIsPositive;

internal bool IsPositive(int int32) { return int32 > 0; }

internal static void CompiledCallLambdaExpressionWithConstructor()

Func<int, bool> isPositive;

if (Functions.cachedIsPositive == null)

Functions.cachedIsPositive = new Func<int, bool>(Functions.Singleton.IsPositive);

isPositive = Functions.cachedIsPositive;

bool result = isPositive.Invoke(1);

}

Here are a few more examples of IIFE:

internal static void ImmediatelyInvokedFunctionExpression()

new Func<int, int, int>((int32A, int32B) => int32A + int32B)(1, 2);

new Action<int>(int32 => int32.WriteLine())(1);

new Func<int, int, int>((int32A, int32B) =>

int sum = int32A + int32B;

return sum;

})(1, 2);

new Func<int, bool>(int32 =>

int32.WriteLine();

return int32> 0;

})(1);

new Action<int>(int32 =>

int32.WriteLine();

Trace.Flush();

})(1);

}

In some other strongly-typed functional languages, like F#, Haskell, etc. the type information can be omitted at design time and inferred at compile time the type information is also not needed for dynamic functional languages, like JavaScript. IIFE was commonly used in JavaScript to modularize or isolate code in the function scope, until the ECMAScript 2015 standard introduces module and block scope to JavaScript.

Closure

Similar to local function, anonymous function also supports closure for capturing free variable:

internal static void AnonymousFunctionWithClosure()

int free = 1; // Outside anonynmous function.

new Action(() =>

int local = 2; // Inside anonymous function.

(local + free).WriteLine();

})(); // 3

}

Its compilation is also similar to local function. The difference is, C# compiler generates closure structure for local function, while it generates closure class for anonymous function, which requires heap allocation and can be a performance overhead. The above code is compiled to:

[CompilerGenerated]

private sealed class Closure1

public int Free;

internal void Add()

int local = 2;

(local + this.Free).WriteLine();

internal static void CompiledAnonymousFunctionWithClosure()

int free = 1;

Closure1 closure = new Closure1() { Free = free };

closure.Add(); // 3

}

Similar to local function, the closure of anonymous function may also introduce the same performance pitfall of memory leak. Closure must be used with caution for anonymous function too, whenever an anonymous function may live longer than the execution of outer function.

Expression bodied function member

C# 6.0 and 7.0 introduce expression bodied function member of type. It enables anonymous function’s lambda operator and expression body syntactic sugar to simplify all kinds of named functions, including instance method, static method, extension method, as well as static constructor, constructor, conversion operator, operator overload, getter only property, property getter, property setter, getter only indexer, indexer getter, indexer setter. It also works for local function:

internal partial class Data

private int value;

static Data() => MethodBase.GetCurrentMethod().Name.WriteLine(); // Static constructor.

internal Data(int value) => this.value = value; // Constructor.

~Data() => MethodBase.GetCurrentMethod().Name.WriteLine(); // Finalizer.

internal bool InstanceEquals(Data other) => this.value == other?.value; // Instance method.

internal static bool StaticEquals(Data @this, Data other) => @this?.value == other?.value; // Static method.

public static Data operator +(Data data1, Data data2) => new Data(data1.value + data2.value); // Operator overload.

public static explicit operator int(Data value) => value.value; // Explicit conversion operator.

public static implicit operator Data(int value) => new Data(value); // Implicit conversion operator.

internal int ReadOnlyValue => this.value; // Getter only property.

internal int ReadWriteValue

get => this.value; // Property getter.

set => this.value = value; // Property setter.

internal int this[long index] => throw new NotImplementedException(); // Getter only indexer.

internal int this[int index]

get => throw new NotImplementedException(); // Indexer getter.

set => throw new NotImplementedException(); // Indexer setter.

internal event EventHandler Created

add => MethodBase.GetCurrentMethod().Name.WriteLine(); // Event accessor.

remove => MethodBase.GetCurrentMethod().Name.WriteLine(); // Event accessor.

internal int GetValue()

int LocalFunction() => this.value; // Local function.

return LocalFunction();

}

This syntax works for extension method and interface explicit implementation too:

internal internal static partial class DataExtensions

internal static bool ExtensionEquals(Data @this, Data other) => @this?.ReadOnlyValue == other?.ReadOnlyValue; // Extension method.

internal partial class Data : IEquatable<Data>

bool IEquatable<Data>.Equals(Data other) => this.value == other?.value; // Explicit interface implementation.

}

The expression body is just a syntactic sugar. Its compilation has no difference from the statement body with curly bracket.

C# functional programming in-depth (5) Delegate: Function type, instance and group

Wed, 05 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

C#’s delegate is an important feature to make function first class citizen just like object. C# is a strongly-typed language, where any value and any expression that evaluates to a value has a type. In C#, a value can be an object, which has type represented by class; a value can also be a function, which has type represented by delegate type. Delegate type can be instantiated to represent a single function instance, or a group of functions.

Delegate type as function type

C# supports delegate since the beginning to represent function. In many C# documents, the term “a delegate” could be confusing. It can refer to a delegate type, or an instance of some delegate type. So, this book always uses specific term “delegate type” or “delegate instance” when mentioning a delegate.

Function type

Function’s type can be determined by its input types (0, 1, or more types in specific order) and an output type (void which can be represented by System.Void structure, or another type). This book uses notation input types –> output type for function type. For example, the simplest function type has no input and no output. In another word, it is parameterless, and it returns void. Such function type is denoted () –> void. In C#, a delegate type is defined like a method signature with the delegate keyword prepended:

// () -> void

internal delegate void FuncToVoid();

FuncToVoid can be viewed as an alias of function type () –> void. The following functions are all parameterless, and returning void:

namespace System.Diagnostics

public sealed class Trace

public static void Close();

public static void Flush();

public static void Indent();

}

These functions are all of function type () –> void; in another word, these functions are all of FuncToVoid type.

The following delegate type represents the string –> void function type, which accepts a string parameter, and returns void:

// string -> void

internal delegate void FuncStringToVoid(string @string);

The following functions are all of FuncStringToVoid type:

namespace System.Diagnostics

public sealed class Trace

public static void TraceInformation(string message);

public static void Write(string message);

public static void WriteLine(string message);

}

These functions’ parameter names are different from the delegate type definition. In C#/.NET, parameter names are ignored when the compiler identifies function types, only parameter types, their order, and return type matter.

The following delegate type represents the () –> int function type that is parameterless, and returns int:

// () -> int

internal delegate int FuncToInt32();

The following functions are all of FuncToInt32 type:

namespace System.Runtime.InteropServices

public static class Marshal

public static int GetExceptionCode();

public static int GetHRForLastWin32Error();

public static int GetLastWin32Error();

}

And the following delegate type represents the (string, int) –> int function type that accepts a string parameter, then an int parameter, and returns int:

// (string, int) -> int

internal delegate int FuncStringInt32ToInt32(string @string, int int32);

It is the type of the following functions (Again, the parameter names are ignored.):

namespace System.Globalization

public static class CharUnicodeInfo

public static int GetDecimalDigitValue(string s, int index);

public static int GetDigitValue(string s, int index);

}

The following delegate type represents the string –> bool function type that accepts a string parameter, and returns bool:

// string –> bool

internal delegate bool FuncStringToBoolean(string @string);

The following functions are all of FuncStringToBoolean type:

namespace System

[DefaultMember("Chars")]

public sealed class String : IEnumerable<char>, IEnumerable, IComparable, IComparable<String>, IConvertible, IEquatable<String>

public static bool IsNullOrEmpty(String value);

public static bool IsNullOrWhiteSpace(String value);

public bool Contains(String value);

public bool Equals(String value);

public bool StartsWith(String value);

public bool EndsWith(String value);

}

Generic delegate type

Above FuncToInt32 represents the () –> int function type that is parameterless and return int. Similarly, for parameterless functions returning bool, string, or object result, the following delegate types can be defined:

// () -> bool

internal delegate bool FuncToBoolean();

// () -> string

internal delegate string FuncToString();

// () -> object

internal delegate object FuncToObject();

More similar function type definitions can go forever for functions parameterless with different output types. Since C# 2.0 introduces generics, these types can be represented with one single generic delegate type with a type parameter of any name, like TResult:

// () -> TResult

internal delegate TResult Func<TResult>();

Similar to generic interface/class/structure/method syntax, here type parameter TResult is defined in angle brackets following type name, and it is used as the return type. It is just a placeholder to be specified with concrete type argument later. When TResult is int, Func<int> represents the () –> int function type, which is equivalent to FuncToInt32, and Func<bool> is equivalent to FuncToBoolean, Func<string> is equivalent to FuncToString, Func<object> is equivalent to FuncToObject, etc. All the parameterless function types with an output can be represented by Func<TResult>.

Here is another example of generic delegate type:

// (T1, T2) -> TResult

internal delegate TResult Func<T1, T2, TResult>(T1 value1, T2 value2);

The above generic delegate type can represent any function type with 2 parameters and a return result. For example, Func<string, int, int> is equivalent to above FuncStringInt32ToInt32, so the above CharUnicodeInfo.GetDecimalDigitValue and CharUnicodeInfo.GetDigitalValue functions are of Func<string, int, int> type too. The following are more functions with 2 parameters and a return result:

namespace System

public static class Math

// (double, double) -> double

public static double Log(double a, double newBase);

// (int, int) -> int

public static int Max(int val1, int val2);

// (double, int) -> double

public static double Round(double value, int digits);

// (decimal, MidpointRounding) -> decimal

public static decimal Round(decimal d, MidpointRounding mode);

}

These functions’ types can be represented with Func<double, double, double>, Func<int, int, int>, Func<double, int, double> and Func<decimal, MidpointRounding, decimal>.

Unified built-in delegate types

As fore mentioned, delegate types can be defined with duplicate, like Func<int> and FuncToInt32 are equivalent, Func<string, int, int> and FuncStringInt32ToInt32 are equivalent, etc. For example, since .NET Framework 2.0, the following delegate type is provided:

namespace System

// (T, T) -> int

public delegate int Comparison<in T>(T x, T y);

}

The following custom delegate types can be defined too:

// (T, T) -> int

internal delegate int NewComparison<in T>(T x, T y);

// (string, string) -> TResult

internal delegate TResult FuncStringString<TResult>(string value1, string value2);

// (T1, T2) -> int

internal delegate int FuncToInt32<T1, T2>(T1 value1, T2 value2);

// (string, string) -> int

internal delegate int FuncStringStringToInt32(string value1, string value2);

As a result, Func<string, string, int>, Comparison<string>, NewComparison<int>, FuncStringString<int>, FuncToInt32<string, string>, FuncStringStringToInt32 all represent (string, string) –> int function type. They are all equivalent.

Even .NET built-in delegate types can duplicate. For example, .NET Framework 2.0 also provides the following delegate types, which all represent object –> void function type, and are now in .NET Standard:

namespace System.Threading

// object -> void

public delegate void SendOrPostCallback(object state);

// object -> void

public delegate void ContextCallback(object state);

// object -> void

public delegate void ParameterizedThreadStart(object obj);

// object -> void

public delegate void WaitCallback(object state);

// object -> void

public delegate void TimerCallback(object state);

}

To avoid this kind of duplication, since .NET Framework 3.5, 2 series of built-in delegate types are provided to unify all the function types. The following generic Func delegate types can represent any function type with 0 ~ 16 parameters and a return result:

namespace System

// () -> TResult

public delegate TResult Func<out TResult>();

// T -> TResult

public delegate TResult Func<in T, out TResult>(T arg);

// (T1, T2) -> TResult

public delegate TResult Func<in T1, in T2, out TResult>(T1 arg1, T2 arg2);

// (T1, T2, T3) -> TResult

public delegate TResult Func<in T1, in T2, in T3, out TResult>(T1 arg1, T2 arg2, T3 arg3);

// (T1, T2, T3, T4) -> TResult

public delegate TResult Func<in T1, in T2, in T3, in T4, out TResult>(T1 arg1, T2 arg2, T3 arg3, T4 arg4);

// ...

// (T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16) -> TResult

public delegate TResult Func<in T1, in T2, in T3, in T4, in T5, in T6, in T7, in T8, in T9, in T10, in T11, in T12, in T13, in T14, in T15, in T16, out TResult>(T1 arg1, T2 arg2, T3 arg3, T4 arg4, T5 arg5, T6 arg6, T7 arg7, T8 arg8, T9 arg9, T10 arg10, T11 arg11, T12 arg12, T13 arg13, T14 arg14, T15 arg15, T16 arg16);

}

The in/out modifiers for the type parameter specifies that type parameter is variant, which are discussed in detail in the covariant and contravariant chapter. Unfortunately, above Func types cannot represent any function types returning void. Function type Func<void> or Func<System.Void> cannot be compiled, because C# complier does not allow generic’s type argument to be the void keyword or the System.Void type. So, the following generic Action delegate types are provided to represent all function types with 0 ~ 16 parameters, and no return result:

namespace System

// () -> void

public delegate void Action();

// T -> void

public delegate void Action<in T>(T obj);

// (T1, T2) -> void

public delegate void Action<in T1, in T2>(T1 arg1, T2 arg2);

// (T1, T2, T3) -> void

public delegate void Action<in T1, in T2, in T3>(T1 arg1, T2 arg2, T3 arg3);

// (T1, T2, T3, T4) -> void

public delegate void Action<in T1, in T2, in T3, in T4>(T1 arg1, T2 arg2, T3 arg3, T4 arg4);

// ...

// (T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16) -> void

public delegate void Action<in T1, in T2, in T3, in T4, in T5, in T6, in T7, in T8, in T9, in T10, in T11, in T12, in T13, in T14, in T15, in T16>(T1 arg1, T2 arg2, T3 arg3, T4 arg4, T5 arg5, T6 arg6, T7 arg7, T8 arg8, T9 arg9, T10 arg10, T11 arg11, T12 arg12, T13 arg13, T14 arg14, T15 arg15, T16 arg16);

}

For consistency, this book always uses the above Func and Action delegate types to represent function types.

Delegate instance as function instance

Just like object can be instantiated from class, delegate instance can be instantiated from delegate type too. A delegate instance can represent a function, or a group of functions of the same function type.

To represent a single function, a delegate instance can be constructed by passing the specified function to constructor call syntax, which is similar to the syntax of constructing an object:

internal static void InstantiationWithConstructor()

Func<int, int, int> func = new Func<int, int, int>(Math.Max);

int result1 = func(1, 2);

result.WriteLine(); // 2

}

The constructor call syntax can be omitted:

internal static void Instantiation()

Func<int, int, int> func = Math.Max;

int result = func(1, 2);

result.WriteLine(); // 2

}

With this syntax, above paradigm looks functional. Func<int, int, int> is the function type, func variable is the function (instance), and func variable’s value is initialized with the Math.Max function. And naturally, function func can be called. When func is called, Math.Max executes and returns the result.

The above functional paradigm is actually implemented by wrapping imperative object-oriented programming. For each delegate type definition, C# compiler generates a class definition. For example, System.Func<T1, T2, TResult> delegate type is compiled to the following class derived from System.MulticastDelegate class:

public sealed class CompiledFunc<in T1, in T2, out TResult> : MulticastDelegate

public CompiledFunc(object @object, IntPtr method);

public virtual TResult Invoke(T1 arg1, T2 arg2);

public virtual IAsyncResult BeginInvoke(T1 arg1, T2 arg2, AsyncCallback callback, object @object);

public virtual void EndInvoke(IAsyncResult result);

}

And MulticastDelegate derives System.Delegate class:

namespace System

public abstract class Delegate

public object Target { get; }

public MethodInfo Method { get; }

public static Delegate Combine(params Delegate[] delegates);

public static Delegate Combine(Delegate a, Delegate b);

public static Delegate Remove(Delegate source, Delegate value);

public static bool operator ==(Delegate d1, Delegate d2);

public static bool operator !=(Delegate d1, Delegate d2);

// Other members.

public abstract class MulticastDelegate : Delegate

public static bool operator ==(MulticastDelegate d1, MulticastDelegate d2);

public static bool operator !=(MulticastDelegate d1, MulticastDelegate d2);

}

The generated delegate class has an Invoke method, with the same signature as the delegate type itself. The above delegate instantiation code is a syntactic sugar compiled to normal object instantiation, and the function call is also a syntactic sugar compiled to above Invoke method call:

internal static void CompiledInstantiation()

CompiledFunc<int, int, int>func = new CompiledFunc<int, int, int>(null, Math.Max); // object is null for static method.

int result = func.Invoke(1, 2);

result.WriteLine(); // 2

}

The generated Invoke method can be useful along with null conditional operator to call a delegate instance if it is not null:

internal static void Invoke<T>(Action<T> action, T arg)

action?.Invoke(arg); // if (action != null) { action(arg); }

}

Each delegate instance inherits Target/Method properties, and ==/!= operators from Delegate and MulticastDelegate. The following example demonstrates these members work for delegate instance representing static method:

internal static void StaticMethod()

Func<int, int, int> func1 = Math.Max;

int result1 = func1(1, 2); // func1.Invoke(1, 2);;

(func1.Target is null).WriteLine(); // True

MethodInfo method1 = func1.Method;

$"{method1.DeclaringType}: {method1}".WriteLine(); // System.Math: Int32 Max(Int32, Int32)

Func<int, int, int> func2 = Math.Max;

object.ReferenceEquals(func1, func2).WriteLine(); // False

(func1 == func2).WriteLine(); // True

}

As fore mentioned, delegate instance func1 looks like a function and works like a function with syntactic sugar, but it is essentially an instance of a generated delegate class. It has an Invoke method accepting 2 int parameters and returning int. Its Target property returns the underlying object which has the represented method. When the underlying method is a static method, Target returns null. Its Method property returns a MethodInfo instance that represents the underlying method, Math.Max. Then delegate instance func2 is instantiated with the same static method, and apparently it is another different instance from func1, so object.ReferenceEquals returns false. However, func1 and func2 wraps the same underlying static method, and the == operator returns true.

In contrast, the following delegate instances represent instance methods:

internal static void InstanceMethod()

object instance1 = new object();

Func<object, bool> func1 = instance1.Equals;

object.ReferenceEquals(func1.Target, instance1).WriteLine(); // True

MethodInfo method2 = func1.Method;

$"{method2.DeclaringType}: {method2}".WriteLine(); // System.Object: Boolean Equals(System.Object)

object instance2 = new object();

Func<object, bool> func2 = instance2.Equals;

object.ReferenceEquals(func2.Target, instance2).WriteLine(); // True

object.ReferenceEquals(func1, func2).WriteLine(); // False

(func1 == func2).WriteLine(); // False

Func<object, bool> func3 = instance1.Equals;

object.ReferenceEquals(func1, func3).WriteLine(); // False

(func1 == func3).WriteLine(); // True

}

Here func1’s Target property returns instance1, which has the underlying instance method. When 2 delegate instance have the same underlying instance method from the same target, the == operator returns true.

The BeginInvoke and EndInvoke methods are provided for asynchrony:

internal static void TraceAllTextAsync(string path)

Func<string, string> func = File.ReadAllText;

func.BeginInvoke(path, TraceAllTextCallback, func);

internal static void TraceAllTextCallback(IAsyncResult asyncResult)

Func<string, string> func = (Func<string, string>)asyncResult.AsyncState;

string allText = func.EndInvoke(asyncResult);

allText.WriteLine();

}

C# asynchronous programming should follow the async/await pattern introduced in C# 5.0 instead of the above BeginInvoke/EndInvoke pattern. The async/await pattern is discussed in the asynchronous function chapter.

Delegate instance as function group

Besides a single function, delegate instance can also represent a group of function of the same type. The following functions are all of () –> string type:

internal static string A() { return MethodBase.GetCurrentMethod().Name.WriteLine(); }

internal static string B() { return MethodBase.GetCurrentMethod().Name.WriteLine(); }

internal static string C() { return MethodBase.GetCurrentMethod().Name.WriteLine(); }

internal static string D() { return MethodBase.GetCurrentMethod().Name.WriteLine(); }

Functions can be combined to a group with + operator, and function can be removed from a group with - operator:

internal static void FunctionGroup()

Func<string> a = A;

Func<string> b = B;

Func<string> functionGroup1 = a + b;

functionGroup1 += C;

functionGroup1 += D;

string lastResult1 = functionGroup1(); // A B C D

lastResult1.WriteLine(); // D

Func<string> functionGroup2 = functionGroup1 - a;

functionGroup2 -= D;

string lastResult2 = functionGroup2(); // B C

lastResult2.WriteLine(); // C

Func<string> functionGroup3 = functionGroup1 - functionGroup2 + a + A;

string lastResult3 = functionGroup3(); // A D A A

lastResult3.WriteLine(); // A

}

Here functionGroup1 is combination of A + B + C + D. When functionGroup1 is called, the 4 internal functions are called one by one, and functionGroup1’s returns the last combined function’s result “D”. functionGroup2 is functionGroup1 – A – D, which is B + C, so functionGroup2’s return value is “C”. functionGroup3 is functionGroup1 – functionGroup2 + a + A, which is A + B + A + A, so its return result is “A”. Actually, + is compiled to Delegate.Combine call and – is compiled to Delegate.Remove call:

internal static void CompiledFunctionGroup()

Func<string> a = A;

Func<string> b = B;

Func<string> functionGroup1 = (Func<string>)Delegate.Combine(a, b); // = A + B

functionGroup1 = (Func<string>)Delegate.Combine(functionGroup1, new Func<string>(C)); // += C

functionGroup1 = (Func<string>)Delegate.Combine(functionGroup1, new Func<string>(D)); // += D

string lastResult1 = functionGroup1.Invoke(); // A B C D

lastResult1.WriteLine(); // D

Func<string> functionGroup2 = (Func<string>)Delegate.Remove(functionGroup1, a); // = functionGroup1 - A

functionGroup2 = (Func<string>)Delegate.Remove(functionGroup2, new Func<string>(D)); // -= D

string lastResult2 = functionGroup2.Invoke(); // B C

lastResult2.WriteLine(); // C

Func<string> functionGroup3 = (Func<string>)Delegate.Combine((Func<string>)Delegate.Combine((Func<string>)Delegate.Remove(functionGroup1, functionGroup2), a), new Func<string>(A)); // = functionGroup1 - functionGroup2 + a + A

string lastResult3 = functionGroup3(); // A D A A

lastResult3.WriteLine(); // A

}

C# language implements event with delegate instance as function. To keep it simple and consistent, this book always use delegate instance to represent single function in all non-event scenarios.

Event and event handler

C# event implements the observer pattern of object-oriented programming. After learning how delegate instance as group works, it is very easy to understand event from a functional programming perspective – an event member is virtually a delegate instance as function group. The following Downloader type can download string from the specified URI, with a Completed event defined:

internal class DownloadEventArgs : EventArgs

internal DownloadEventArgs(string content) { this.Content = content; }

internal string Content { get; }

internal class Downloader

internal event EventHandler<DownloadEventArgs> Completed;

private void OnCompleted(DownloadEventArgs args)

EventHandler<DownloadEventArgs> functionGroup = this.Completed;

functionGroup?.Invoke(this, args);

internal void Start(string uri)

using (WebClient webClient = new WebClient())

string content = webClient.DownloadString(uri);

this.OnCompleted(new DownloadEventArgs(content));

}

It has a Start method to start downloading. When the downloading is done, Start calls OnCompleted, and OnCompleted raises the Completed event member by calling the Completed event as if it is a delegate instance. The type of event member is EventHandler<TEventArgs> generic delegate type:

namespace System

// (object, TEventArgs) -> void

public delegate void EventHandler<TEventArgs>(object sender, TEventArgs e);

}

So EventHandler<DownloadEventArgs> represents (object, DownloadEventArgs) –> void function type, where the object argument is the Downloader instance which raises the event, and the DownloadEventArgs argument is the event data, which wraps the downloaded string. The Completed event’s handler must be function of the same (object, DownloadEventArgs) –> void type. The following are 2 examples:

// (object, DownloadEventArgs) -> void: EventHandler<DownloadEventArgs> or Action<object, DownloadEventArgs>

internal static void TraceContent(object sender, DownloadEventArgs args)

args.Content.WriteLine();

// (object, DownloadEventArgs) -> void: EventHandler<DownloadEventArgs> or Action<object, DownloadEventArgs>

internal static void SaveContent(object sender, DownloadEventArgs args)

File.WriteAllText(Path.GetTempFileName(), args.Content);

}

Now the += operator can be used to add an event handler function to the event function group, and –= operator can be used to remove the event handler function from the event function group:

internal static void Event()

Downloader downloader = new Downloader();

downloader.Completed += SaveContent;

downloader.Completed += TraceContent;

downloader.Completed -= SaveContent;

downloader.Start("https://weblogs.asp.net/dixin");

}

When the Start method is called, it downloads the string. Once the download is completed, it raises the Completed event, which is virtually calling a function group. So that the event handler function in the group is called. To be accurately understand this mechanism, the Completed event member of type (object, DownloadEventArgs) –> void is compiled to 3 members: a delegate instance field, an add_Completed method, and a remove_Completed method:

internal class CompiledDownloader

private EventHandler<DownloadEventArgs> completedGroup;

internal void add_Completed(EventHandler<DownloadEventArgs> function)

EventHandler<DownloadEventArgs> oldGroup;

EventHandler<DownloadEventArgs> group = this.completedGroup;

do

oldGroup = group;

EventHandler<DownloadEventArgs> newGroup = (EventHandler<DownloadEventArgs>)Delegate.Combine(oldGroup, function);

group = Interlocked.CompareExchange(ref this.completedGroup, newGroup, oldGroup);

} while (group != oldGroup);

internal void remove_Completed(EventHandler<DownloadEventArgs> function)

EventHandler<DownloadEventArgs> oldGroup;

EventHandler<DownloadEventArgs> group = this.completedGroup;

do

oldGroup = group;

EventHandler<DownloadEventArgs> newGroup = (EventHandler<DownloadEventArgs>)Delegate.Remove(oldGroup, function);

group = Interlocked.CompareExchange(ref this.completedGroup, newGroup, oldGroup);

} while (group != oldGroup);

}

The generated delegate instance field is the function group to store the event handler functions. The add_Completed and remove_Completed methods adds and removes event handler functions by calling Delegate.Combine and Delegate.Remove, in a in a thread safe approach. It can be simplified by deleting the Interlocked method calls for thread safety, and representing the (object, DownloadEventArgs) –> void delegate type with the normal unified delegate type Action<object, DownloadEventArgs>. The following code shows the essentials after compilation:

internal class SimplifiedDownloader

private Action<object, DownloadEventArgs> completedGroup;

internal void add_Completed(Action<object, DownloadEventArgs> function)

this.completedGroup += function;

internal void remove_Completed(Action<object, DownloadEventArgs> function)

this.completedGroup -= function;

private void OnCompleted(DownloadEventArgs args)

Action<object, DownloadEventArgs> functionGroup = this.completedGroup;

functionGroup?.Invoke(this, args);

internal void Start(string uri)

using (WebClient webClient = new WebClient())

string content = webClient.DownloadString(uri);

this.OnCompleted(new DownloadEventArgs(content));

internal static void CompiledHandleEvent()

SimplifiedDownloader downloader = new SimplifiedDownloader();

downloader.add_Completed(SaveContent);

downloader.add_Completed(TraceContent);

downloader.remove_Completed(SaveContent);

downloader.Start("https://weblogs.asp.net/dixin");

}

So, the C# event/event handler model is quite straightforward from functional programming perspective. It is all about function type, function, and function group:

· An event is a member of class or structure, as a C#/.NET programming convention, it should be of function type (object, TEventArgs) –> void. If the event is an instance member of a class or structure, the object parameter is the instance of that class or structure which raises the event; if the event is static member, the object parameter should be null. The other TEventArgs parameter should derive from System.EventArgs class, and wraps the information of the event, like the downloaded content of a download complete event, the cursor’s position for a mouse click event, etc.

· As a convention, event member’s type is usually represented by EventHandler<TEventArgs> delegate type, which is equivalent to Action<object, TEventArgs>.

· Compiler generates 3 members for an event member: a field member, which is a delegate instance as function group to store event handler function, along with 2 helper method members to add/remove event handler function.

· An event’s event handler is a function of the same (object, TEventArgs) –> void type.

· To handle an event, use the += operator to add the event handler function to the event function group.

· To raise an event, just call the function group, as a result, all the event handler functions stored in the group are called to handle the event.

This compilation of event member is similar to an auto property member, which can be compiled to a backing field, a getter and a setter. Actually, C# has an event add/remove accessor syntax similar to property getter/setter:

internal class DownloaderWithEventAccessor

internal event EventHandler<DownloadEventArgs> Completed

add { this.Completed += value; }

remove { this.Completed -= value; }

}

The add/remove accessors are compiled to above add_Event/remove_Event helper methods.

Summary

This chapter discusses delegate, an important functional feature of C#. Delegate type represents function type, and .NET Standard provides unified delegate types. Delegate type can be instantiated to represent a single function, or a function group. C#’s event is implemented by delegate instance as function group. Delegate is the foundation of C# functional programming to enable first class function.

C# functional programming in-depth (4) Function input and output

Tue, 04 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

The previous 2 chapters discuss all the named functions in C#. This chapter looks into the input and output features of C# function.

Input by copy vs. input by alias (ref parameter)

In C#, by default, arguments are passed to parameters by making a copy (also called passing by value). In the following example, the InputByCopy function has a Uri parameter and an int parameter. System.Uri is class so it is reference type, and int (System.Int32) is structure so it is value type:

internal static void InputByCopy(Uri reference, int value)

reference = new Uri("https://flickr.com/dixin");

value = 10;

internal static void CallInputByCopy()

Uri reference = new Uri("https://weblogs.asp.net/dixin");

int value = 1;

InputByCopy(reference, value); // Copied.

reference.WriteLine(); // https://weblogs.asp.net/dixin

value.WriteLine(); // 1

}

Here a reference type variable and a value type variable are initialized and passed to InputByCopy as arguments. With the default behaviour, the reference and the value are both copied. Similar to local variable discussed in the C# language basics chapter, for reference type, another reference is created to point to the same instance, while for value type, another instance is allocated with copied data. Then the copied reference and value are passed into InputByCopy function, and mutated by the function. So, the original variables are not impacted.

Similar to ref local variable, parameter with a ref modifier means input by alias without copying (also called passing by reference):

internal static void InputByAlias(ref Uri reference, ref int value)

reference = new Uri("https://flickr.com/dixin");

value = 10;

internal static void CallInputByAlias()

Uri reference = new Uri("https://weblogs.asp.net/dixin");

int value = 1;

InputByAlias(ref reference, ref value); // Not copied.

reference.WriteLine(); // https://flickr.com/dixin

value.WriteLine(); // 10

}

This time, a reference type variable and a value type variable are initialized and passed to InputByAlias. InputByAlias’s arguments are just aliases of the original variables. When InputByAlias is called to mutate the aliases, the original variables are mutated as well.

Input by immutable alias (in parameter)

To prevent function from mutating the input by alias, in modifier can be used for the parameter since C# 7.2:

internal static void InputByImmutableAlias(in Uri reference, in int value)

reference = new Uri("https://flickr.com/dixin"); // Cannot be compiled.

value = 10; // Cannot be compiled.

}

The in modifier for parameter is similar to the ref readonly modifiers for the local variable. Similarly, inside the function, trying to mutate the immutable alias causes compile time error.

Output parameter (out parameter) and out variable

C# also supports output parameter, which has a out modifier. The output parameter is also input by alias just like ref parameter:

internal static bool OutputParameter(out Uri reference, out int value)

reference = new Uri("https://flickr.com/dixin");

value = 10;

return false;

internal static void CallOutputParameter()

Uri reference;

int value;

OutputParameter(out reference, out value); // Not copied.

reference.WriteLine(); // https://flickr.com/dixin

value.WriteLine(); // 10

}

The difference is, the ref parameter is input of the function, so a variable must be initialized before passed to the ref parameter. The out parameter can be viewed as output of the function, so a variable is not required to be initialized before being passed to the out parameter. Instead, out parameter must be initialized inside the function.

C# 7.0 introduces a convenient syntactic sugar called out variable, so that a variable can be declared inline without initialization when it is passed to an out parameter. The above example can be simplified as:

internal static void OutVariable()

OutputParameter(out Uri reference, out int value); // Not copied.

reference.WriteLine(); // https://flickr.com/dixin

value.WriteLine(); // 10

}

The out variable declaration is compiled to normal variable declaration

Discard out variable

Since C# 7.0, if a out argument is not needed, it can be simply discarded with special character _. This syntax works with local variable too.

internal static void Discard()

bool result = OutputParameter(out _, out _);

OutputParameter(out _, out _);

_ = OutputParameter(out _, out _);

}

Parameter array

Array parameter with params modifier is called parameter array:

internal static int Sum(params int[] values)

// Compiled to: Sum([ParamArray] int[] values)

int sum = 0;

foreach (int value in values)

sum += value;

return sum;

}

The params modifier is compiled to System.ParamArrayAttribute. When calling above function, any number of arguments can be passed to its parameter array, and, of course, array can be passed to parameter array too:

internal static void CallSum()

int sum1 = Sum();

int sum2 = Sum(0);

int sum3 = Sum(0, 1, 2, 3, 4);

int sum4 = Sum(new[] { 0, 1, 2, 3, 4 });

}

When passing argument list to parameter array, the argument list is always compiled to non-null array:

internal static void CompiledCallSum()

int sum1 = Sum(Array.Empty<int>());

int sum2 = Sum(new int[] { 0 });

int sum3 = Sum(new int[] { 0, 1, 2, 3, 4 });

int sum4 = Sum(new int[] { 0, 1, 2, 3, 4 });

}

When function has multiple parameters, the parameter array must be the last:

internal static void MultipleParameters(bool required1, int required2, params string[] optional) { }

Positional argument vs. named argument

By default, when calling a function, each argument must align with the parameter’s position. C# 4.0 introduces named argument, which enables specifying parameter name when passing an argument. Both positional argument and named argument can be used to call function:

internal static void PositionalArgumentAndNamedArgument()

InputByCopy(null, 0); // Positional arguments.

InputByCopy(reference: null, value: 0); // Named arguments.

InputByCopy(value: 0, reference: null); // Named arguments.

InputByCopy(null, value: 0); // Positional argument followed by named argument.

InputByCopy(reference: null, 0); // Named argument followed by positional argument.

}

When a function is called with positional arguments, the arguments must align with the parameters. When a function is called with named arguments, the named arguments can be in arbitrary order. And when using positional and named arguments together, before C# 7.2, positional arguments must be followed by named arguments. Since C# 7.2, when all arguments are in correct position, then named argument can precede positional argument. At compile time, all named arguments are compiled to positional arguments. The above InputByCopy calls are compiled to:

internal static void CompiledPositionalArgumentAndNamedArgument()

InputByCopy(null, 1);

InputByCopy(null, 1);

InputByCopy(null, 1);

InputByCopy(null, 1);

InputByCopy(null, 1);

}

If the named arguments are evaluated with inline function call, the order of evaluation is the same as their appearance:

internal static void NamedArgumentEvaluation()

InputByCopy(reference: GetUri(), value: GetInt32()); // Call GetUri then GetInt32.

InputByCopy(value: GetInt32(), reference: GetUri()); // Call GetInt32 then GetUri.

internal static Uri GetUri() { return default; }

internal static int GetInt32() { return default; }

When the above InputByCopy calls are compiled, local variable is generated to ensure the arguments are evaluated in the specified order:

internal static void CompiledNamedArgumentEvaluation()

InputByCopy(GetUri(), GetInt32()); // Call GetUri then GetInt32.

int value = GetInt32(); // Call GetInt32 then GetUri.

InputByCopy(GetUri(), value);

}

In practice, this syntax should be used with cautious because it can generate local variable, which can be slight performance hit. This tutorial uses named argument syntax frequently for readability:

internal static void NamedArgument()

UnicodeEncoding unicodeEncoding1 = new UnicodeEncoding(true, true, true);

UnicodeEncoding unicodeEncoding2 = new UnicodeEncoding(

bigEndian: true, byteOrderMark: true, throwOnInvalidBytes: true);

}

Required parameter vs. optional parameter

By default, function parameters require arguments. C# 4.0 also introduces optional parameter, with a default value specified:

internal static void OptionalParameter(

bool required1, char required2,

int optional1 = int.MaxValue, string optional2 = "Default value.",

Uri optional3 = null, Guid optional4 = new Guid(),

Uri optional5 = default, Guid optional6 = default) { }

The default value for optional parameter must be compile time constant, or default value of the type (null for reference type, or default constructor call for value type, or default expression). If a function has both required parameters and optional parameters, the required parameters must be followed by optional parameters. Optional parameter is not a syntactic sugar. The above function is compiled as the following CIL:

.method assembly hidebysig static

void OptionalParameter(

bool required1,

char required2,

[opt] int32 optional1,

[opt] string optional2,

[opt] class [System]System.Uri optional3,

[opt] valuetype [mscorlib]System.Guid optional4,

[opt] class [System]System.Uri optional5,

[opt] valuetype [mscorlib]System.Guid optional6

) cil managed

.param [3] = int32(2147483647) // optional1 = int.MaxValue

.param [4] = "Default value." // optional2 = "Default value."

.param [5] = nullref // optional3 = null

.param [6] = nullref // optional4 = new Guid()

.param [7] = nullref // optional5 = default

.param [8] = nullref // optional6 = default

.maxstack 8

IL_0000: nop

IL_0001: ret

}

And function with optional parameters can be called with the named argument syntax too:

internal static void CallOptionalParameter()

OptionalParameter(true, '@');

OptionalParameter(true, '@', 1);

OptionalParameter(true, '@', 1, string.Empty);

OptionalParameter(true, '@', optional2: string.Empty);

OptionalParameter(

optional6: Guid.NewGuid(), optional3: GetUri(), required1: false, optional1: GetInt32(),

required2: Convert.ToChar(64)); // Call Guid.NewGuid, then GetUri, then GetInt32, then Convert.ToChar.

}

When calling function with optional parameter, if the argument is not provided, the specified default value is used. Also, local variables can be generated to ensure the argument evaluation order. The above Optional calls are compiled to:

internal static void CompiledCallOptionalParameter()

OptionalParameter(true, '@', 1, "Default value.", null, new Guid(), null, new Guid());

OptionalParameter(true, '@', 1, "Default value.", null, new Guid(), null, new Guid());

OptionalParameter(true, '@', 1, string.Empty, null, new Guid(), null, new Guid());

OptionalParameter(true, '@', 1, string.Empty, null, new Guid(), null, new Guid());

Guid optional6 = Guid.NewGuid(); // Call Guid.NewGuid, then GetUri, then GetInt32, then Convert.ToChar.

Uri optional3 = GetUri();

int optional1 = GetInt32();

OptionalParameter(false, Convert.ToChar(64), optional1, "Default value.", optional3);

}

Caller information parameter

C# 5.0 introduces caller information parameters. System.Runtime.CompilerServices.CallerMemberNameAttribute, System.Runtime.CompilerServices.CallerFilePathAttribute, System.Runtime.CompilerServices.CallerLineNumberAttribute can be used for optional parameters to obtain the caller function name, caller function file name, and line number:

internal static void TraceWithCaller(

string message,

[CallerMemberName] string callerMemberName = null,

[CallerFilePath] string callerFilePath = null,

[CallerLineNumber] int callerLineNumber = 0)

Trace.WriteLine($"[{callerMemberName}, {callerFilePath}, {callerLineNumber}]: {message}");

}

When calling function with caller information parameters, just omit those arguments:

internal static void CallTraceWithCaller()

TraceWithCaller("Message.");

// Compiled to:

// TraceWithCaller("Message.", "CompiledCallTraceWithCaller", @"D:\Data\GitHub\Tutorial\Tutorial.Shared\Functional\InputOutput.cs,", 219);

}

At compile time, these omitted arguments are generated for the caller.

Output by copy vs. output by alias

By default, function return result by making a copy (also called returning by value). Which is similar to input by copy by default. The following functions retrieve the first item from the specified array:

internal static int FirstValueByCopy(int[] values)

return values[0];

internal static Uri FirstReferenceByCopy(Uri[] references)

return references[0];

}

When they return the first item to the caller, they return a copied of the reference or value. When the returned item mutates, the item in the array is not impacted:

internal static void OutputByCopy()

int[] values = new int[] { 0, 1, 2, 3, 4 };

int firstValue = FirstValueByCopy(values); // Copy of values[0].

firstValue = 10;

values[0].WriteLine(); // 0

Uri[] references = new Uri[] { new Uri("https://weblogs.asp.net/dixin") };

Uri firstReference = FirstReferenceByCopy(references); // Copy of references[0].

firstReference = new Uri("https://flickr.com/dixin");

references[0].WriteLine(); // https://weblogs.asp.net/dixin

}

C# 7.0 introduces output by alias (also called returning by reference). Similar to input by alias, returned result with a ref modifier is not copied:

internal static ref int FirstValueByAlias(int[] values)

return ref values[0];

internal static ref Uri FirstReferenceByAlias(Uri[] references)

return ref references[0];

}

The above functions can be called with the ref modifier to avoid copying. This time, when the returned alias mutates, the item in the array mutates too:

internal static void OutputByAlias()

int[] values = new int[] { 0, 1, 2, 3, 4 };

ref int firstValue = ref FirstValueByAlias(values); // Alias of values[0].

firstValue = 10;

values[0].WriteLine(); // 10

Uri[] references = new Uri[] { new Uri("https://weblogs.asp.net/dixin") };

ref Uri firstReference = ref FirstReferenceByAlias(references); // Alias of references[0].

firstReference = new Uri("https://flickr.com/dixin");

references[0].WriteLine(); // https://flickr.com/dixin

}

Output by immutable alias

To prevent caller from modifying the returned alias, ref can be used with the readonly modifier since C# 7.2:

internal static ref readonly int FirstValueByImmutableAlias(int[] values)

return ref values[0];

internal static ref readonly Uri FirstReferenceByImmutableAlias(Uri[] references)

return ref references[0];

}

Now the above functions can be called with ref modifier, and the readonly modifier is required for the returned alias. Apparently, trying to mutate the returned immutable alias causes error at compile time:

internal static void OutputByImmutableAlias()

int[] values = new int[] { 0, 1, 2, 3, 4 };

ref readonly int firstValue = ref FirstValueByImmutableAlias(values); // Immutable alias of values[0].

#if DEMO

firstValue = 10; // Cannot be compiled.

#endif

Uri[] references = new Uri[] { new Uri("https://weblogs.asp.net/dixin") };

ref readonly Uri firstReference = ref FirstReferenceByImmutableAlias(references); // Immutable alias of references[0].

#if DEMO

firstReference = new Uri("https://flickr.com/dixin"); // Cannot be compiled.

#endif

}}

Summary

This chapter dicusses many input and output features of C# functions, including input by copy, input by alias (ref parameter), input by immutable alias (in parameter), output parameter, out variable, parameter array, named argument, optonal parameter, caller ingormation parameter, output by copy, output by alias, and output by immutable alias.

C# functional programming in-depth (3) Local Function and Closure

Mon, 03 Jun 2019 00:00:00 GMT

[LINQ via C# series]

[C# functional programming in-depth series]

The previous chapter discussed named function. There is one more special kind of named function. local function. Local function can be nested in another function, and it supports an important feature closure, the ability access variable outside the local function itself.

Local function

C# 7.0 introduces local function, which allows defining and calling a named, nested function inside a function. Unlike a local variable, which has to be used after being defined, a local function can be called before or after it is defined:

internal static void MethodWithLocalFunction()

void LocalFunction() // Define local function.

MethodBase.GetCurrentMethod().Name.WriteLine();

LocalFunction(); // Call local function.

internal static int PropertyWithLocalFunction

get

LocalFunction(); // Call local function.

void LocalFunction() // Define local function.

MethodBase.GetCurrentMethod().Name.WriteLine();

LocalFunction(); // Call local function.

return 0;

}

Local function is a syntactic sugar. Its definition is compiled to normal method definition, and its call is compiled to method call. For example, the above method with local function is compiled to:

[CompilerGenerated]

internal static void CompiledLocalFunction()

MethodBase.GetCurrentMethod().Name.WriteLine();

internal static void CompiledMethodWithLocalFunction()

CompiledLocalFunction();

}

Besides method members, local function can also be nested inside local function:

internal static void LocalFunctionWithLocalFunction()

void LocalFunction()

void NestedLocalFunction() { }

NestedLocalFunction();

LocalFunction();

}

Anonymous function can have local function as well (Anonymous function is discussed in an individual chapter):

internal static Action AnonymousFunctionWithLocalFunction()

return () => // Return an anonymous function of type Action.

void LocalFunction() { }

LocalFunction();

};

}

Unlike other named functions, local function does not support ad hoc polymorphism (overload). The following code cannot be compiled:

internal static void LocalFunctionOverload()

void LocalFunction() { }

void LocalFunction(int int32) { } // Cannot be compiled.

}

Local function is useful to encapsulate code execution in a function. For example, the following binary search function encapsulate the main algorithm in a local function, and execute it recursively:

internal static int BinarySearchWithLocalFunction<T>(this IList<T> source, T value, IComparer<T> comparer = null)

int BinarySearch(

IList<T>localSource, T localValue, IComparer<T>localComparer, int startIndex, int endIndex)

if (startIndex > endIndex) { return -1; }

int middleIndex = startIndex + (endIndex - startIndex) / 2;

int compare = localComparer.Compare(localSource[middleIndex], localValue);

if (compare == 0) { return middleIndex; }

return compare > 0

? BinarySearch(localSource, localValue, localComparer, startIndex, middleIndex - 1)

: BinarySearch(localSource, localValue, localComparer, middleIndex + 1, endIndex);

return BinarySearch(source, value, comparer ?? Comparer<T>.Default, 0, source.Count - 1);

}

C# local function supports closure, so above local function can be further simplified, which is discussed later in this chapter.

Local function is also useful with asynchronous function and generator function to isolate the asynchronous execution and deferred execution, which are discussed in the asynchronous function and LINQ to Objects chapters.

Closure

In object-oriented programming, it is “perfectly nature normal thing” for a type’s method member to use local variable and field member:

internal class Closure

int field = 1; // Outside function Add.

internal void Add()

int local = 2; // Inside function Add.

(local + field).WriteLine(); // local + this.field.

}

Here in Closure type, its method accesses data inside and outside its definition. Similarly, local function can access variable inside and outside its definition as well:

internal static void LocalFunctionWithClosure()

int free = 1; // Outside local function Add.

void Add()

int local = 2; // Inside local function Add.

(local + free).WriteLine();

Add();

}

Here free is a variable defined by outer function and is outside the local function. In C# it can be accessed by both the outer function and the local function. It is the local variable of the outer function, and it is called free variable of the local function. In another word, for a local function, if a variable is neither its local variable, nor its parameter, then this variable is its free variable. Free variable is also called outer variable, non-local variable, or captured variable. This capability for local function to access a free variable, is called closure. C# closure is also a syntactic sugar. The above example is compiled to a closure structure:

[CompilerGenerated]

[StructLayout(LayoutKind.Auto)]

private struct Closure1

public int Free;

[CompilerGenerated]

private static void CompiledAdd(ref Closure1 closure)

int local = 2;

(local + closure.Free).WriteLine();

internal static void CompiledLocalFunctionWithClosure()

int free = 1;

Closure1 closure = new Closure1() { Free = free };

CompiledAdd(ref closure);

}

C# compiler generates:

· A closure structure to capture the free variable as field.

· A normal method member definition to represent the local function, with a closure parameter. In its body, the reference to free variable is compiled to reference to closure’s field.

· A normal method member call with a closure argument, whose field is initialized with the free variable. The instantiated closure is passed to the generated method member as alias to avoid copying the closure instance, since it is a structure. Function input as alias is discussed in the function input and output chapter.

So, C# compiler implements closure, a functional feature, by generating object-oriented code.

The above binary search function’s local function accesses the source to search, target value, and comparer through parameter. With closure, the local function does not need these parameters. It can directly access them as free variable:

internal static int BinarySearchWithClosure<T>(this IList<T> source, T value, IComparer<T> comparer = null)

int BinarySearch(int startIndex, int endIndex)

if (startIndex > endIndex) { return -1; }

int middleIndex = startIndex + (endIndex - startIndex) / 2;

int compare = comparer.Compare(source[middleIndex], value);

if (compare == 0) { return middleIndex; }

return compare > 0

? BinarySearch(startIndex, middleIndex - 1)

: BinarySearch(middleIndex + 1, endIndex);

comparer = comparer ?? Comparer<T>.Default;

return BinarySearch(0, source.Count - 1);

}

It is compiled to the same closure structure and method member pattern:

[CompilerGenerated]

[StructLayout(LayoutKind.Auto)]

private struct Closure2<T>

public IComparer<T> Comparer;

public IList<T> Source;

public T Value;

[CompilerGenerated]

private static int CompiledLocalBinarySearch<T>(int startIndex, int endIndex, ref Closure2<T> closure)

if (startIndex > endIndex) { return -1; }

int middleIndex = startIndex + (endIndex - startIndex) / 2;

int compare = closure.Comparer.Compare(closure.Source[middleIndex], closure.Value);

if (compare == 0) { return middleIndex; }

return compare <= 0

? CompiledLocalBinarySearch(middleIndex + 1, endIndex, ref closure)

: CompiledLocalBinarySearch(startIndex, middleIndex - 1, ref closure);

internal static int CompiledBinarySearchWithClosure<T>(IList<T> source, T value, IComparer<T> comparer = null)

Closure2<T> closure = new Closure2<T>()

Source = source,

Value = value,

Comparer = comparer

};

return CompiledLocalBinarySearch(0, source.Count - 1, ref closure);

}

As demonstrated above, when the local function has multiple free variables, it still has 1 closure parameter. The closure structure defines multiple fields to capture all free variable and pass to the local function as parameter.

Free variable mutation

Apparently, free variable is variable and it can mutate. When mutation happens, the accessing local functions can be impacted. In the previous example, if the free variable mutates, the local function apparently outputs different sum of local variable and free variable:

internal static void FreeVariableMutation()

int free = 1;

void Add()

int local = 2;

(local + free).WriteLine();

Add(); // 3

free = 3; // Free variable mutates.

Add(); // 5

}

Sometimes, this can be source of problems.

internal static void FreeVariableReference()

List<Action> localFunctions = new List<Action>();

for (int free = 0; free < 3; free++) // free is 0, 1, 2.

void LocalFunction() { free.WriteLine(); }

localFunctions.Add(LocalFunction);

} // free is 3.

foreach (Action localFunction in localFunctions)

localFunction(); // 3 3 3 (instead of 0 1 2)

}

In this case, the for loop has 3 iterations. In the first iteration, free is 0, a local function is defined to output free’s value, and the local function is stored to a function list. In the second iteration, free becomes 1, a local function is repeatedly defined to write free’s value, and stored in function list, and so on. Later, when calling these stored local functions, they do not output 0, 1, 2, but 3, 3, 3. The reason is, the 3 iterations of for loop share the same free variable, when the for loop is done, the free’s value becomes 3. Then, calling these 3 functions outputs the latest value of outer for 3 times, so it is 3, 3, 3. The compiled code is more intuitive. Notice the local function is compiled to a method member of closure structure, since it is stored:

[CompilerGenerated]

private struct Closure3

public int Free;

internal void LocalFunction() { this.Free.WriteLine(); }

internal static void CompiledFreeVariableReference()

List<Action> localFunctions = new List<Action>();

Closure3 closure = new Closure3();

for (closure.Free = 0; closure.Free < 3; closure.Free++) // free is 0, 1, 2.

localFunctions.Add(closure.LocalFunction);

} // closure.Free is 3.

foreach (Action localFunction in localFunctions)

localFunction(); // 3 3 3 (instead of 0 1 2)

}

This can be resolved by capture a snapshot of shared free’s current value in each iteration, and store it in another variable that does not mutate:

internal static void CopyFreeVariableReference()

List<Action> localFunctions = new List<Action>();

for (int free = 0; free < 3; free++) // free is 0, 1, 2.

int copyOfFree = free;

// When free mutates, copyOfFree does not mutate.

void LocalFunction() { copyOfFree.WriteLine(); }

localFunctions.Add(LocalFunction);

} // free is 3. copyOfFree is 0, 1, 2.

foreach (Action localFunction in localFunctions)

localFunction(); // 0 1 2

}

In each iteration of for loop, free is copied to copyOfFree. copyOfFree is not shared cross the iterations and does not mutate. When the for loop is done, 3 local function calls output the values of 3 snapshot values 0, 1, 2.. Above code is compiled to:

[CompilerGenerated]

private sealed class Closure4

public int CopyOfFree;

internal void LocalFunction() { this.CopyOfFree.WriteLine(); }

internal static void CompiledCopyFreeVariableReference()

List<Action> localFunctions = new List<Action>();

for (int free = 0; free < 3; free++)

Closure4 closure = new Closure4() { CopyOfFree = free }; // free is 0, 1, 2.

// When free changes, closure.CopyOfFree does not change.

localFunctions.Add(closure.LocalFunction);

} // free is 3. closure.CopyOfFree is 0, 1, 2.

foreach (Action localFunction in localFunctions)

localFunction(); // 0 1 2

}

Each iteration of the for loop instantiate an independent closure, which captures copyOfFree instead of free. When the for loop is done, each closure’s instance method is called to output its own captured value.

Performance

C# closure provides great convenience to enable local function to directly access free variable. Besides allocating structure on stack, closure may also lead to performance pitfall, because it generates closure structure with reference to the accessed free variable, and that reference is not intuitive at all for developers at design time. The following is a closure example with large free variable:

internal static partial class LocalFunctions

private static Action persisted;

internal static void FreeVariableLifetime()

byte[] tempLargeInstance = new byte[0x_7FFF_FFC7]; // Temp variable of large instance, Array.MaxByteArrayLength is 0x_7FFF_FFC7.

// ...

void LocalFunction()

// ...

int length = tempLargeInstance.Length; // Reference to free variable.

// ...

length.WriteLine();

// …

// ...

LocalFunction();

// ...

persisted = LocalFunction; // Reference to local function.

}

Here temp is a large instance of byte array. It is a temporary local variable of the outer function, and free variable of the local function. It is not explicitly stored to any other variable or field, and supposed to have a short lifetime along with the execution of outer function. However, this temporary variable cannot be garbage collected after the execution of outer function and local function. The reason is, the local function is stored to a static field, and persisted to a long lifetime, so that its free variable should has the same lifetime. The problem is not intuitive at design time. At compile time, the following closure is generated:

[CompilerGenerated]

private sealed class Closure5

public byte[] TempLargeInstance;

internal void LocalFunction()

int length = this.TempLargeInstance.Length;

length.WriteLine();

internal static void CompiledFreeVariableLifetime()

byte[] tempLargeInstance = new byte[0X7FFFFFC7];

Closure5 closure = new Closure5() { TempLargeInstance = tempLargeInstance };

closure.LocalFunction();

persisted = closure.LocalFunction;

// closure's lifetime is bound to persisted, so is closure.TempLargeInstance.

}

The large array is captured as a field of the closure structure, which is expected since it is the free variable of the local function. Since the local function is stored, it is also compiled to be a method member of the closure structure. Here comes the problem. When the outer function stores the local function to the static field, it actually instantiates the closure, and stores closure’s instance method to the static field. Since the instance method’s lifetime is persisted, the entire closure instance is persisted with a long lifetime. after the execution of outer function and local function, the closure along cannot be deallocated, with its field of large array not able to be garbage collected, which causes memory leak issue. To fix the issue, consider a different design where local function is not persisted, or local function does not access large instance through free variable.

Multiple local functions in one function may share the same closure, which may also lead to memory leak. The following example’s problem is even more obscure:

internal static Action SharedClosure()

byte[] tempLargeInstance = new byte[0x_7FFF_FFC7];

void LocalFunction1() { int length = tempLargeInstance.Length; }

LocalFunction1();

bool tempSmallInstance = false;

void LocalFunction2() { tempSmallInstance = true; }

LocalFunction2();

return LocalFunction2; // Return a function of Action type.

internal static void CallSharedClosure()

persisted = SharedClosure(); // Returned LocalFunction2 is persisted.

}

Here LocalFunction2 only accesses free variable tempSmallInstance, and has nothing to do with tempLargeInstance. However, if SharedClosure is called and the returned LocalFunction2 is persisted, tempLargeInstance is still leaked and cannot be garbage collected. Again, the problem is invisible at design time, but intuitive at compile time:

[CompilerGenerated]

private struct Closure6

public byte[] TempLargeInstance;

internal void LocalFunction1() { int length = this.TempLargeInstance.Length; }

public bool TempSmallInstance;

internal void LocalFunction2() { this.TempSmallInstance = true; }

internal static Action CompiledSharedClosure()

Closure6 closure = new Closure6();

closure.TempLargeInstance = new byte[0x_7FFF_FFC7];

closure.LocalFunction1();

closure.TempSmallInstance = false;

closure.LocalFunction2();

return closure.LocalFunction2; // Return a function of Action type.

}

C# compiler can generate one shared closure structure for multiple local functions and their free variables. If one local function is persisted, the shared closure is persisted, along with all captured free variables of all local functions. Besides a different design not persisting local function or not accessing large free variable, another possible improvement is to separate local functions to different lexical scopes:

internal static Action SeparatedClosures()

{ // Lexical scope has its own closure.

byte[] tempLargeInstance = new byte[0x_7FFF_FFC7];

void LocalFunction1() { int length = tempLargeInstance.Length; }

LocalFunction1();

bool tempSmallInstance = false;

void LocalFunction2() { tempSmallInstance = true; }

LocalFunction2();

return LocalFunction2; // Return a function of Action type.

}

C# compiler generates an individual closure for each lexical scopes, so the above 2 local function are compiled to 2 separated closures. If the returned LocalFunction2 is persisted, only tempSmallInstance is persisted along with LocalFunction2’s closure.

So, whenever a local function may live longer than the execution of outer function, free variable must be used with caution. Other languages supporting closure in similar way, like JavaScript, etc., has the same pitfall.

Static local function

C# 8.0 introduces static local function. Closure is disabled when static keyword is used to define local function.

Summary

This chapter explains what is local function, how to define and use it, and its underlying compilation. This chapter also discusses local function’s important feature closure, including the concept, usage, compilation, and analyzed possible performance pitfall of memory leak.

CodingOnWheels.com

C# Functional Programming In-Depth (14) Asynchronous Function

[LINQ via C# series]

[C# functional programming in-depth series]

Task, Task<TResult> and asynchrony

Named async function

Awaitable-awaiter pattern

Async state machine

Runtime context capture

Generalized async function output type and async method builder

ValueTask<TResult> and performance

Anonymous async function

Asynchrnous sequence: IAsyncEnumerable<T>

async using declaration: IAsyncDispose

Summary

Functional Programming and LINQ via C#

Acclaim

Contents at a Glance

Table of Contents

Understanding (all) JavaScript module formats and tools

IIFE module: JavaScript module pattern

IIFE: Immediately invoked function expression

Import mixins

Revealing module: JavaScript revealing module pattern

CJS module: CommonJS module, or Node.js module

AMD module: Asynchronous Module Definition, or RequireJS module

Dynamic loading

AMD module from CommonJS module

UMD module: Universal Module Definition, or UmdJS module

UMD for both AMD (RequireJS) and native browser

UMD for both AMD (RequireJS) and CommonJS (Node.js)

ES module: ECMAScript 2015, or ES6 module

ES dynamic module: ECMAScript 2020, or ES11 dynamic module

System module: SystemJS module

Dynamic module loading

Webpack module: bundle from CJS, AMD, ES modules

Babel module: transpile from ES module

Babel with SystemJS

TypeScript module: Transpile to CJS, AMD, ES, System modules

Internal module and namespace

Conclusion

Port Microsoft Concurrency Visualizer SDK to .NET Standard and NuGet

TransientFaultHandling.Core: Retry library for .NET Core/.NET Standard

Introduction

Document

Source

NuGet installation

Backward compatibility with Enterprise Library

How to use the APIs

Object-oriented APIs from Enterprise Library

New functional APIs: single function call for retry

New fluent APIs for retry

Old XML configuration for retry

New XML/JSON/INI configuration for retry

TransientFaultHandling.Data.Core: SQL Server support

History

Category Theory via C# (8) Advanced LINQ to Monads

[FP & LINQ via C# series]

[Category Theory via C# series]

IO monad

State monad

Exception monad

Reader monad

Writer monad

Continuation monad

Category Theory via C# (7) Monad and LINQ to Monads

[FP & LINQ via C# series]

[Category Theory via C# series]

Monad

LINQ to Monads and monad laws

Built-in IEnumerable<> monad

Monad law and Kleisli composition

Kleisli category

Monad pattern of LINQ

Monad vs. monoidal/applicative functor

More LINQ to Monads

Category Theory via C# (6) Monoidal Functor and Applicative Functor

[FP & LINQ via C# series]

[Category Theory via C# series]

Monoidal functor