I think you may be right but there were loads of countries that did boycott and ironically China was one as I remember the commotion on the news about it .
Alan Wells won the 100m gold in Moscow 1980 after the USA pulled out. IIRC there was pressure on the athletes to pull out, and a number of our teams didn't go - equestrian, hockey and yatching teams all boycotted. Taiwan boycotted in protest of China insisting they were called "Chinese Taipei", then China boycotted it too. Some Irish and NZ athletes competed under the Olympic banner.