(este post es una explicación de la teoría. dejo aquí otro post con el detalle práctico de como usar la temperature, top_k y top_p a efectos prácticos)

Explicación más en detalle de cómo obtienen los LLMs las probabilidades para generar el siguiente token

Sampling

En cada posición el modelo tiene una bolsa con miles de tokens posibles. Sampling es el proceso de no coger siempre el token más probable, si no de orientarlo a coger respuestas con determinadas características.

Si el modelo escogiera siempre el token con más probabilidades, obtendríamos respuestas aburridas y repetitivas.

Logits

Para generar el siguiente token, una red neuronal calcula primero los vectores de logits, donde cada logit corresponde a un valor posible. El tamaño de estos vectores de logits es tan grande como el vocabulario completo del modelo.

(representación de vectores de logits)

flowchart LR
	N1["What's your favorite color?"]:::note --> Z
	Z --> A1 --> A
	Z --> B1 --> B
	Z --> C1 --> C
	Z --> D1 --> D
	
	Z["Neural network"]
    A["a"]
    A1["(-0.5)"]
    B["green"]
    B1["(0.7)"]
    C["red"]
    C1["(0.5)"]
    D["the"]
    D1["(-1.2)"]
    
    classDef note fill:none,stroke:none,color:#777;    

Los logits NO representan probabilidades ya que no suman 1 y pueden incluso ser negativos (la probabilidades no pueden). Para convertir logits a probabilidades se usa una Softmax layer

Temperature

La temperatura es una constante que se aplica a los logits antes de la transformación de la Softmax layer. Se usa para ajustar la creatividad del modelo y redistribuir la probabilidad de los valores. Una temperatura más alta hace que el modelo sea más creativo ya que aumenta las posibilidades de elegir tokens menos probables.

xychart-beta
  title "Temperatura vs Probabilidad"
  x-axis "Temperatura (T)" [0.1, 0.2, 0.5, 1, 2, 5]
  y-axis "Probabilidad" 0 --> 1
  line "P(token1)" [0.9999546, 0.9933071, 0.8807971, 0.7310586, 0.6224593, 0.5498340]
  line "P(token2)" [0.0000454, 0.0066929, 0.1192029, 0.2689414, 0.3775407, 0.4501660]

Ejemplos de temperaturas:

• Low (0.2-0.3): El modelo es cauto y elige las palabras más probables. Output factual y predecible.
• Medium (0.5-0.7): Un mix de confiabilidad y engagement
• High (0.9-1.0): Toma riesgos y es impredecible

Read More

I’ve had issues with EF Core when operating with multiple threads and with multiple calls at the same time.

The most important things to check are:

1. The DbContext is not being shared between calls or threads
2. All classes which have the context inyected must be scoped (not singleton)
3. If working with async methods, you need to await calls

I have the following service

public class PersonService(AppDbContext _context)
{
	public async Task<Person> GetPerson(string id)
	{
		return await context.Persons.Find(id);
	}
}

which I may configure as follows

// if I inject it as singleton, this would cause exceptions on multiple calls
services.AddSingleton<IPersonService, PersonService>

// we have to inject it as scoped so it creates a context new for each call
services.AddScoped<IPersonService, PersonService>

.NET offers several cache types. I’m going to explore here IMemoryCache which stores data in the memory of the web server. It’s simple but not suitable for distributed scenarios.

first of all we need to register the services

builder.Services.AddMemoryCache();

GetOrCreateAsync

here’s how you can inject and use it, without manipulating the cache itself

public class PersonService(IMemoryCache _cache)
{
	private const string CACHE_PERSON_KEY = "PersonService:GetPerson:";

	public async Task<Person> GetPerson(string id)
	{
		return await _cache.GetOrCreateAsync(CACHE_PERSON_KEY + id, async entry =>
		{
			entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(5);
			return await GetPersonNoCache(id);
		});
	}

	public async Task<Person> GetPersonNoCache(string id)
	{
		// do operations to get a person here
	}
}

Read More

Split tasks in its minimum definition and estimate those minimum tasks by Optimistic (O) - Most Likely (M) - Pessimistic (P).

With those estimations we do PERT distribution and then add those estimations

(O+(4xM)+P)/6

Example

Task: Migrate x database Minimum tasks:

• migrate service 1 to y database
• migrate service 2 to y database
• migrate connector to use y database
• test changes in test env

Then we estimate those tasks

Reference(s)

https://www.knowledgehut.com/blog/project-management/three-point-estimating

The following is an example where we need to call and await an external API multiple times inside an iteration.

I’m using myFakeAPI from postman for this example and one of their Car response look like this

public class CarResponse
{
	public CarDto Car { get; set; }
}

public class CarDto
{
	public int Id { get; set; }
	public string Car { get; set; }
	public string Car_Model { get; set; }
	public string Car_Color { get; set; }
	public int Car_Model_Year { get; set; }
	public string Car_Vin { get; set; }
	public string Price { get; set; }
	public bool Availability { get; set; }
}

Then this is the method which does call and mapping

private async Task<CarResponse> ExecuteCall(string id)
{
	string combinedUrl = URL + id;

	using var response = await _httpClient.GetAsync(combinedUrl);
	response.EnsureSuccessStatusCode();

	string json = await response.Content.ReadAsStringAsync();
	return JsonConvert.DeserializeObject<CarResponse>(json);
}

Control

This is the control version where we launch and await the tasks one at a time

// DON'T DO THIS
private async Task<List<CarResponse>> Control()
{
	List<CarResponse> carList = [];
	foreach (string id in _idsList)
	{
		CarResponse singleCar = await ExecuteCall(id);
		carList.Add(singleCar);
	}
	return carList;
}

Task.WhenAll()

It’s the most simple one - all tasks are launched at the same time. It’s ideal when we don’t have limits as we have no control over the simultaneous number of calls

// simple but what if we'd have +100 calls?
private async Task<List<CarResponse>> TaskWhenAll()
{
	var getCarsTask = _idsList.Select(ExecuteCall);
	var cars = await Task.WhenAll(getCarsTask);
	return cars.ToList();
}

Parallel.ForEachAsync()

This gives us the most control over number of parallel calls. It’s more complex.

private async Task<List<CarResponse>> ParallelForEachAsync()
{
	// this is a secure collection for multiple threads
	var carsBag = new ConcurrentBag<CarResponse>();
	var options = new ParallelOptions { MaxDegreeOfParallelism = 5 };

	await Parallel.ForEachAsync(_idsList, options, async (id, ct) =>
	{
		CarResponse car = await ExecuteCall(id);
		carsBag.Add(car);
	});
	return carsBag.ToList();
}

Example on how to use generics in C#

public class AnimalService(IConnectorService _service)
{
	public async Task<List<T>> GetAnimals<T> (List<string> ids, string query)
	{
		List<T> results = [];
		var request = new ConnectorRequest
		{
			query = query,
			ids = ids
		};
		response = await _service.Execute(request);
		if((response?.result?.Count ?? 0) > 0)
		{
			results = JsonConvert.DeserializeObject<List<T>>(response.result);
		}
		return results;
	}
}

JsonConvert.SerializeObject

I use this to serialize full objects to log them with all their properties

InputModel x = // ...
log.LogInfo($"doing x. input: {JsonConvert.SerializeObject(x)}");

JsonProperty and NullValueHandling

This is useful for cases where we need to modify the given properties of a class we serialize and give back, but for any reason we don’t want to change the internal structure or naming.

With NullValueHandling we may omit in the JSON a variable in case it’s null.

public class House
{
	public List<Window> windows { get; set; };
	
	[JsonProperty("builtInGarage"), NullValueHandling = NullValueHandling.Ignore]
	public Garage garage { get; set; }; 
}

This is how to retrieve headers from any call.

// how to retrieve a mandatory header
if(Request.Headers.TryGetValue("mandatory-header", out var mandatoryHeader))
{
	// this one may be either filled or empty
	string optionalHeader = Request.Headers["optional-header"];
	var result = await _service.DoWork(mandatoryHeader, optionalHeader)
}
else 
{
	// log error as mandatory-header isn't included in the call
}

The core for asynchronous programming are the objects Task and Task<T>. Both of them are compatible with the keywords async and await.

First of all we need to identify if the code’s I/O-bound or CPU-bound.

• the code’s limited for external operations and waits for something a lot of time. Examples of this are DDBB calls, or a server’s response. In this case we have to use async/await to free the thread while we wait
• the code does a CPU-intensive operation. Then we move the work to another thread using Task.Run() so we don’t block the main thread.

async code vs parallel code

(!) Asynchronous code is not the same as parallel code (!)

• In async code you are trying to make your threads do as little work as possible. This will keep your app responsibe, capable to serve many requests at once and scale well.
• In parallel code you do the opposite. You use and keep a hold on a thread to do CPU-intensive calculations

async code

The importante of async programming is that you choose when to wait on a task. This way, you can start other tasks concurrently

In async code, one single thread can start the next task concurrently before the previous one completes.
(!) async code doesn’t cause additional threads to be created because an async method doesn’t run on its own thread. (!) It runs on the current synchronization context and uses time on the thread only when the method is active.

parallel code

For parallelism you need multiple threads where each thread executes a task, and all of those tasks are executed at the same time

Read More

In Azure AD B2C you’ve two ways to provide identity UX:

• user flows - predefined, built-in, configurable policies so you can create sign-up/in and policy editing UX in minutes.
• custom policies - enable you to create your own user journeys for complex identity experience scenarios.

User Flows

For the most common identity tasks. Things like:

• Account types used for sign-in, such as social accounts or local accounts.
• Attributes to be collected from the consumer.
• MFA
• User interface customization
• Set of claims in a token your app receives

Custom Policies

They’re config files that define the behaviour of your Azure AD B2C tenant UX. They can be fully edited by an identity developer to complete many different tasks.

It’s fully configurable and policy-driven.

• Federate with other identity providers
• Third party MFA
• Collect any user input
• Integrate with external systems using REST API communication

Each user journey is defined by a policy. You can build as many or as few policies as you need.

Defined by several XML files that refer to each other in a hierarchical chain.

Starter Pack

The starter pack comes with pre-built policies.

• LocalAccounts - Enables the use of local accounts only.
• SocialAccounts - Enables the use of social (or federated) accounts only.
• SocialAndLocalAccounts - Enables the use of both local and social accounts. Most of our samples refer to this policy.
• SocialAndLocalAccountsWithMFA - Enables social, local, and multi-factor authentication options.

Each starter pack includes:

• a base file that contains most definitions. To help with troubleshooting and long-term maintenance of your policies, try to minimize the number of changes you make to this file.
• an extension file holds the unique config changes for your tenant. This file is derived from the base file. Use this to add new functionality or override existing functionality. e.g. to federate with new identity providers.
• a relying party (RP) file is the single task-focused file that’s invoked directly by the relying party application, such as web, mobile or desktop app. Each unique task such as sign-up/in, password reset etc requires its own relying party policy file. This file is derived from the extensions file. You can add more relying party policies. (e.g. delete my account, change a phone number…)

The basics

Claim

A claim provides temporary storage of data during an Azure AD B2C policy execution. It can store info about the user, such as first name, last name… or any other claim obtainer from the user or other systems. The claim schema is the place where you declare your claims.

When a policy runs, B2C sends and receives claims to and from internal and external parties, then sends a subset of these claims to your relying party app as part of the token.

• Claims are saved, read, or updated against the directory user object.
• Claims are received from an external identity provder.
• Claims are sent or received using a custom REST API service.
• Data is collected as claims from the user during sign-up or edit profile flows.

Claims transformation are predefined functions to convert a given claim into another one, evaluate a claim or set a claim value.

User Journey

They allow you to define business logic with path through which users will follow to gain access to your application. The user is taken through this journey to retrieve the claims that are to be presented to your app.

A user journey is built from a sequence of orchestration steps.

How you can add orchestration steps to social and local account starter pack.

Orchestration Steps

A user must reach the last step to acquire a token. Orchestration steps can be conditionally executed based on preconditions. After an step completes, B2C stores the outputted claims in the claims bag. This bag can be used by any further orchestration steps in the user’s journey.

Reference(s)

https://docs.microsoft.com/en-us/azure/active-directory-b2c/user-flow-overview
https://docs.microsoft.com/en-us/azure/active-directory-b2c/custom-policy-overview

Azure AD B2C is Identity as a Service. It allows you to login into apps or use Twitter, Google or other authenticators. They can sign up or sign in. OpenID, OAuth, SAML.

Customers get to use their preferred accounts to sign up or create a regular user / pwd. You can customize the login experience.

App Types

This is best for the following types:

• Server-based web apps work perfectly. They use OpenID Connect for all UX.
• Mobile apps. They use OAuth2 auth code flow.
• Web services and APIs. OAuth2.

This doesn’t work together with Azure AD B2C.

• Daemons.
• Web API chains.

Configure Azure Active Directory B2C

Create new resource -> B2C -> Create new tenant -> enter general information

Notice for the initial domain name, the full domain will be *.onmicrosoft.com

Once a tenant is provisioned, click to manage it. A new tab opens, this is because the B2C tenant is separate from your Azure subscription’s tenant. Your Azure subscription has an Azure Active Directory Tenant associated with it and this new B2C one is distinct and separate. It’s its own directory.

Inside we have the following modules or tabs:

• Applications: websites or web APIs or Apps
• Identity provider: lists all identity providers such as Facebook or Github and your users may log in through them.
• User attributes: you can see all metadata you collect from users.
• Users: you can see and edit all users who have created accounts.
• User flows: Heart and soul of Azure AD B2C as it guides users through the sign in or sign up, or password resets.
• Audit Logs: Authentication of various apps or administrative events.

Link Azure AD B2C Tenant

Before being able to use the B2C instance, you also have to link the B2C directory back to the main subscription. This makes it a service for you to use.

Create new resource -> Azure Active Directory B2C -> Create -> Link existing Azure AD B2C Tenant to my Azure subscription -> select the one you just created -> put it into a resource group

User flow

The flows are reusable. They controls when social accounts can be used to sign in.
The attributes to collect from the user. (you can collect standard or custom ones). To use MFA or not.
User interface customization. Information inside the token that’s returned from B2C. It can be used across B2C apps. They’re entirely reusable.

Azure AD B2C Application - it’s not your web or API. It’s models apps that have authentication added. It makes sure only your users can sign-in.

Redirect URI - where to redirect response after the user’s been authenticated. As a test, put https://jwt.ms to see what’s inside the JWT token.

Built-in User Flows

There’re 3 recommended to have, but you can create custom ones:

• Sign-in/up.
• Profile editing.
• Password reset.

Everything that controls the journeys of the users is located under Policies.

Social Providers

Social network provider are called identity providers (IdP). In order to use them, you have to configure them first. You will get a key which you enter into B2C and the authentication will happen seamlessly.

B2C Application Deep Dive

A B2C app models the real world. Every real-world app needs a B2C app.

reply URL: URL where responses will be returned to your webapp. App ID: unique ID that identifies your B2C app.
They obey standards like OAuth 2.0 or OpenID Connect.
When you want to interact with it, you specify your user flow.

Everything from Sign in to get Resource token happens inside Azure, not your app.

Tokens

They’re the way that B2C uses to transmit claims about a user calling Apps. They’re all JWT tokens.

ID token: contains claims user to identify user. (e.g. user’s object ID in AD). You must always validate it. Access token: Claims used to identify API permissions. Validate it. Refresh token: It should be a black box to your app. Both others tokens expire.

How server-based apps authentication works with B2C. Underlying authentication protocol is OpenID Connect.

Custom Policies

Individual steps into a user’s journey:

They’re XML files
have a Schema definition (which will be used to return the tokens)
Content definition - Things like how to render pages. Technical profiles - they’re the endpoints and how to communicate with identity providers. Orchestration - steps that are contained within the custom policies

Policy Files

In order to work with custom policies we’ll need to handle three files:

• base file - contains most definitions. make minimal changes. you shouldn’t ever have to change this file. Starting point that every B2C tenant would use. Contains the common elements for everything.
• Extensions file - unique config for tenant. Make changes here that overrides anything in the base file. These changes apply to the entire tenant.
• Relying party file - single task focused that’s invoked by app. This is the file that your app invokes. Use this file to make any final tweaks to the user’s journey.

Reference(s)

User flows and custom policies in Azure Active Directory B2C - Azure AD B2C | Microsoft Learn