Data Masking Sequence

Yes, within the Cribl Stream Mask function, masking rules are applied sequentially. 
You can test this yourself by having two Match and Replace examples.

• Regex: [A-z] /g → Replace: ‘1’
• Regex: [A-z] /g → Replace: ‘2’

After the first Masking Rule is evaluated, all letters will become 1 so the second rule when applied will do nothing as there are no remaining alphabetical characters to match.

For your second question about high complexity regex, firstly, you can test efficiency of regex expressions in certain tools to optimize the amount of searching and backtracking. Writing efficient regex is always desirable.
Secondly, when setting up the function, you can apply Mask rules to specific fields. This ensures you are only evaluating what you are interested in.

Finally, when thinking more broadly, since the mask function can be quite CPU intensive, arranging other functions before the Mask one can make sense. For example, if I’m using an Eval statement to drop certain fields, there’s no point masking then in a function step before dropping them in a subsequent one. Similarly if I’m planning on doing conditional drops in a pipeline, you would want to place this function ahead of masking (Unless you are relying on the output of the masking step to determine what should be dropped).