We currently run LiveCycler ES4 (v11) for the ReaderExtensions website.
We have purchased AEM 6.3 as a license. Could we install just the Reader Extension application without installing the full AEM core server?
no you cannot
you will have to install AEM and then the forms add on package . The forms add on package will give the Reader Extension capabilities in addition to lots of other capabilities