What does out of memory error mean?

Java Virtual Machine (which runs Tomcat which runs XWiki) has run out of heap space and cannot allocate anything anymore. Objects of a Java program live in the heap.

Is it related to MySQL packet size?

No

Why does this error happen?

Because application needs more memory than JVM is allowed to use.

How is that related to attachments?

The way XWiki handles attachments is, well, not perfect yet.

Somewhat related thread is http://lists.xwiki.org/pipermail/users/2007-November/009362.html

So what can you do?

a) Use -Xms and -Xmx java parameters to increase heap size. You can do that by passing these values to CATALINA_OPTS and/or JAVA_OPTS during the container startup process. Something like...

export CATALINA_OPTS="-Xmx512m -Xms512m"
export JAVA_OPTS="-Xmx512m -Xms512m"

b) Implement and contribute to XWiki effective attachment handling.

What are the -Xms and -Xmx params?

-Xms tells java the initial heap size, -Xmx controls max size. The latter is of special interest in this case and can be increased.