If Beowulf is just fiction, then what do the battles, and the monsters that Beowulf fights, represent? What might they symbolize in his world?

Beowulf, a poem created by a Christian monk in the midlands of England most likely sometime between the eighth and ninth centuries CE, is indeed a fictional construct. But, as your question implies, it reflects the very real experience of Beowulf's society, as it confronts both man-created and supernatural dangers, forces that constantly seek to destroy mankind.
The battles central to the poem (i.e., the fights with Grendel, Grendel's mother, and the dragon) and the numerous digressions in which other battles are recalled (like the Finnsburg Fragment and the story of King Heremod) are both symbols of a way of life and actual accounts of the warfare that occupied this society most of the time. Because warfare is endemic in this society, the value of a king lies in his ability to wage war successfully and then distribute the wealth gained through war to his most accomplished and loyal followers. But winning battles is secondary to the achievement of personal glory, and, even as an old king with no need to prove his abilities, Beowulf, just before he confronts the dragon, tells his men:

I dared manya fight in my youth; again as an oldprotector of my people, I'll seek out conflict,and earn some glory, if this mankillerwill come out to me from his earth-hall. (2.2512–16)

Battles, which are a fact of life for Beowulf's Scandinavian warrior society, lead to glory, and even in this quasi-Christian poem, glory and fame are constantly beckoning to the ambitious warrior, especially if he also aspires to kingship—like Scyld Scefing and, later, Beowulf, whose epitaph is "That was a good King!"
Grendel, Grendel's mother, and the dragon represent the supernatural forces arrayed against mankind's peace and happiness. Grendel, whom the Christian poet identifies as a descendant of Cain, has been brought up with "eotens and elves and Orkneys," all of whom are supernatural beings and enemies of mankind. Grendel is angry because, as Cain's kinsman, he cannot approach God and share in mankind's joys, so his response is to kill Hrothgar's (and later Beowulf's) men in Hrothgar's mead-hall, a proxy for God's throne that Grendel can invade. Grendel's supernatural nature is reflected in the fact that "no blade can kill him," which is why Beowulf can only fight him in hand-to-hand combat.
A far more dangerous supernatural being is Grendel's mother, who is animated by both hatred of mankind and the desire to avenge her son's death. She, like her son and the dragon, is part of the natural world that constantly poses a danger to Beowulf's society. When the men are not fighting each other, they are fighting the land and the sea. In the Old English poem The Seafarer, the speaker identifies the three primary causes of death for his society: old age, sickness, and warfare. Grendel's mother, a more accomplished and powerful being than Grendel, almost defeats Beowulf:

[Beowulf] went down then, stunned, the strongest of fighters . . .defeat closed in.Then she sat on the hall-guest, and drew her dagger,broad and bloodstained; she'd avenge her son. (2.1543–46)

Beowulf only saves himself by spotting a blade on the cave wall, "good and elegant, giants' work," that is strong enough to kill Grendel's mother. Her death, fortunately, closes the ring on Beowulf's early struggles with monsters—all of whom are supernatural (beginning with the sea monsters Beowulf and Breca battled) but essentially represent the natural world, which is a constant danger to mankind.
The dragon, of course, is a stock Scandinavian folk-tale enemy of mankind, and, like Grendel and Grendel's mother, it is yet another supernatural being who cannot be reasoned with. Unlike men, who can be made whole by the payment of money if they are harmed in war or civil strife, the dragon is only interested in revenge when the thief steals a cup from its horde, and Beowulf's kingdom is in its sights. The dragon episode is a necessary element in Beowulf's life because the fight allows Beowulf to represent his people in another struggle against violent nature and, more important, to leave his sedentary life of peace with a demonstration of his bravery. This leads, of course, to more personal glory for him—which is the be-all and end-all goal of any Scandinavian warrior king.