External Affairs Minister said prime ministers from Congress gave away Indian fishermen's rights despite legal views to the contrary