In order to understand the usage of these three words, you must know that one of them is a noun (cancellation) and the other two are verb forms and/or participles (canceled and canceling).
CANCELED
As a past tense form, it means that the action was completed in the past.
- He canceled the meeting.
As a participle, it can be part of a verb tense or an adjective.
- He has canceled the meeting.
- The canceled meeting will be rescheduled.
CANCELING
As part of a continuous verb form, it indicates current or continuing action.
- He is canceling the meeting as we speak.
- He was canceling the meeting when his boss called.
As a gerund (a verb acting as a noun), it can serve anywhere a noun can.
- Canceling a meeting takes a lot of effort. (subject)
- He avoided canceling the meeting. (direct object)
- He solved the issue by canceling the meeting. (object of preposition)
CANCELLATION
This word is only a noun. It can never be a verb or an adjective or a gerund. It describes the act of canceling something. It generally is used as a subject. Using it as a direct object or object of preposition is rarely done.
- Cancellation will take a long time. (subject)
Knowing which form to use is simply a matter of understanding grammar and the different parts of speech. By taking the time to study that you can greatly improve your ability to use English correctly.