What are the rules to join sentences without using "and", "so" or "but"?

"And," "so," or "but" are coordinating conjunctions, along with "for," "nor," "or," and "yet." Used with a comma, they connect two sentences (independent clauses).
If you are not using a coordinating conjunction such as "and," "so," or "but" to connect your sentences, you have several options, but the most straightforward one is to use a semi-colon. You can also use a colon or an em dash. The one piece of punctuation you cannot and must not use is a comma. When you connect two independent clauses (sentences) with a weak little comma, that creates what is known as a comma splice, a very common grammatical error.
Let's look at an example. Say you have this as the original sentence:
Mary went to store, and on the way she saw a ghost!
You decide you want to get rid of the "and." You could write the following:
Mary went to the store; on the way, she saw a ghost! That uses a semi-colon.
Or you could write this: Mary went to the store: on the way, she saw a ghost! That uses a colon.
Or you could put it this way: Mary went to the store—on the way, she saw a ghost!
All of these forms of punctuation create enough of a barrier between the independent clauses to be acceptable.
However, you could not write: Mary went to the store, on the way, she saw a ghost! That is a comma splice. It is a big no-no.