1. Create calc element named random_number and use the code below with Only Calc on Save = Yes.
$ids = '{tablename___id_raw}';

$rn = JFactory::getDBO();
$rn->setQuery("SELECT `random_number` FROM `tablename` WHERE `id` = '$ids'");
$rn = $rn->loadResult();

$range = '10'; // Numbering range between 1 to 10.

$db = JFactory::getDBO();
$db->setQuery("SELECT `random_number` FROM `tablename`"); // Find existed value and put in array.
$db = $db->loadColumn();

if (empty(ids)) { // Return if new form if (count($db) >= $range) { return ''; } else { do { $rand = rand(1, $range); } while (in_array($rand, $db)); return $rand; }
} else {
return $rn;
}
2. Use the code below on php form plugin with Process script = OnBeforeProcess to stop the form process.
$rn = '{tablename___random_number}';

if ($rn == '') {
  return false;
}