We understand that SmoothDamp doesn't work in that particular case. But since there is a lot of other projects relying on this method to works, changing this behavior could be problematic for those projects.

Unity SmoothDamp method is based on the book Game Programming Gems 4 Chapter 1.10 which has the same behavior and problem as Unity's SmoothDamp.

The best solution we can provide is to write your own version of smooth damp for your project if the one provided by Mathf doesn't work as expected for you.

You can use this code as an example:

public static float SmoothDamp(float current, float target, ref float currentVelocity, float smoothTime, float maxSpeed, float deltaTime)
{
// Based on Game Programming Gems 4 Chapter 1.10
smoothTime = Mathf.Max(0.0001F, smoothTime);
float omega = 2F / smoothTime;

float x = omega * deltaTime;
float exp = 1F / (1F + x + 0.48F * x * x + 0.235F * x * x * x);
float change = current - target;
float originalTo = target;

// Clamp maximum speed
float maxChange = maxSpeed * smoothTime;
change = Mathf.Clamp(change, -maxChange, maxChange);
target = current - change;

float temp = (currentVelocity + omega * change) * deltaTime;
currentVelocity = (currentVelocity - omega * temp) * exp;
float output = target + (change + temp) * exp;

return output;
}