In the administrative console click:
Servers > Server Types > WebSphere application servers > SERVER_NAME > Server Infrastructure > Java and process management > Process definition > Java Virtual Machine
Set Maximum heap size field to desired value. Restart the server to bring in effect the value you set. Recommended value for maximum heap size by IBM is:
MaximumHeapSize = TotalRAM / 2 / NumberOfServers
JVM maximum heap size can be also set by scripting through wsadmin tool (heap size is in MB):
AdminTask.setJVMMaxHeapSize('-serverName YOUR_SERVER_NAME -nodeName YOUR_NODE_NAME -maximumHeapSize DESIRED_HEAP_SIZE')
In our experience 1536 MB is enough for testing purposes on WPS+Monitor or WPS+Fabric combinations.