I'd be more worried that your gold bars weren't actually in your bankers vault anymore. Look what happened when Germany tried to repatriate their gold from the U.S. Federal Reserve.
They didn't "try", they are doing it, partially, piecemeal.
Which may be because the Fed didn't actually have all that gold available, nicely separate in a vault labelled "Germany", but even if that is true, the case is rather more complex than just "The Fed embezzled the gold that Germany gave them for safekeeping". Because Germany never gave that gold - they never actually had it. It's in fact the German export surpluses from the time before the USA went off the gold standard.
