Walton family witnessed a slump of $9.4 billion in their net worth and this decline pushed the family from rank 17 to 20 in 2006 list of billionaires.
Children of Sam Walton, founder of Wal-Mart founder, saw their fall in fortune this year.
The family ranked No. 10 through No. 13 on last year's list, ranging from oldest son S Robson Walton's $18.2 billion to daughter Helen Walton's $18.0 billion.
The death of Sam Walton's son, John, in a plane crash also pushed daughter-in-law Christy into the No. 17 position.