Assam Chief Minister Himanta Biswa Sarma had alleged on Thursday that his West Bengal counterpart was only concerned about Bengali-speaking Muslims.