Found a solution.
Add (ajaxRequestInitiate)='ajaxRequestInitiateEvent($event)' to the component and edit the jsonData there.
ajaxRequestInitiate(event: AjaxRequestInitiateEventArgs){
if (event.JsonData['action'] == 'Download' && this.pdfViewer.serverActionSettings.download == 'SaveDocument') {
event.JsonData.document = 'PATH TO DOCUMENT';
}
}