The task manager endpoint is just to start (initiate) new processes. Once you've initiated a new process, Workspace will show you the tasks whether you have a Task Manager endpoint or not.
The purpose of Workspace is to initiate processes and show you the tasks that belong to you. If you don't want to initiate processes and don't want to see the tasks, you may as well not use the interface at all.
If you still really want to use the interface, for what ever reason, you'll have to customize it using the source code and remove the things you don't want.