Food laws as prescribed in the Torah were an obvious mark of Jewish identity in the first century CE. The issue of table-fellowship in Galatians was not one of complete abandonment of all the laws regarding table-fellowship. It appears that the "men from James" wanted stricter adherence to the laws concerning food and ritual purity. As an increasing number of Gentiles entered the church, the more important became the issue of the identity of the people of God. Food laws figured into that identity.
The limits of table-fellowship were different from community to community. They were generally more relaxed in Hellenistic Judaism. Devout Jews, however, basically avoided table-fellowship with Gentiles. In demanding stricter devotion to the food and purity laws, the "men from James" were building a wall between Jewish Christians and Gentile Christians. Unless the Gentile believers Judaized there could be little or no table-fellowship between Jewish and Gentile Christians.
For Paul this was inconsistent with the gospel, for it denied the Gentiles full membership in the church. The covenant privileges remained in the hands of ethnic Israel. The Gentile Christians, even as believers, remained "sinners," which for Paul missed the point of justification by faith.