I have changed the memory settings (AddVMOption -Xmx1768M) in datamodeler64.conf but it does not change anything. Why?

Because you must remove the settings from product.conf first. As long as you have them in product.conf those settings are follower.

So either

• change product.conf or
• remove the memory settings from product.conf and change the datamodeler.conf / datamodeler64.conf file.

My personal opinion is that datamodeler.conf / datamodeler64.conf is better place for changes but both choices work..