This is a very confusing question because the whole internet is filled with different spellings of this word Om, and amusingly enough, all the spellings appear correct and pointing to the same thing. Yet, a popular search phrase on Google suggests that users are confused about the veracity of a particular spelling because they want to be consistent in the usage of that very spelling.
On Google, a lot of users search for this phrase: ‘Is it Om or Aum’
Well, in this article, or a discussion, as I would like to call it, we will ponder upon this question. I have an additional spelling ‘Oṁ’, which is also used if you are exposed to the IAST system of writing Sanskrit in roman transliteration. A lot of users from foreign countries prefer reading in IAST system because they are not yet equipped to read Devanāgari.
I must confess in the beginning itself that I am not the authority on this subject–this discussion is based on my personal understanding. I will point out wherever I feel we could use more help by someone knowledgeable in this subject.
Okay, so here are my observations:
To understand the spellings, we need to understand the structure of this word, which is often represented by this symbol: ॐ
We refer to Nārāyaṇopaniṣat (नारायणोपनिषत्), to know the authentic structure of this sound. The Upanishad says:
अकार उकार मकार इति।
The sound of ॐ consists the sounds of ‘A’, ‘U’, and ‘M’.
So, it becomes: A+U+M
Apparently, if you write it as AUM, it looks good.
However, if you refer to the Devanāgari script and read the spelling of this sound instead of its symbolic form, you will notice that it is written as either as ‘ओम्’ or ‘ओं’, the latter is due to the rules of Sandhi (Rules of joining in Sanskrit).
Here, I must point out that I don’t know why it is half ‘M‘ and not full ‘M‘.
So, if you are using, reading, writing the IAST system, you will be writing it like ‘Om’ or ‘Oṁ’.
If you know IAST system, you must have come across the sound of ‘O’, which constitutes the sounds of ‘A’ and ‘U’.
The problem appears when users start learning the reverse system–they begin to recognize the Devanāgari script and start practicing. So, when they reverse-translate Aum, they end up writing it like this: औम्, which is, of course, wrong, because it has a combination of sounds: A+A+U.
So, though any spelling would do if you know the sound of this symbol, yet it would be safe to write it as ‘Om’ instead of ‘Aum’ because later on it could become confusing and end up making you commit errors. It would also be consistent according to the IAST system.